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ABSTRACT 


This  technical  report  summarizes  the  image  understanding  and  image 
processing  research  activities  performed  by  the  University  of  Southern 
California  during  the  period  of  1 September  1975  to  31  March  1976  under 
Contract  No.  F- 336 15-76-C- 1203  with  the  Advanced  Research  Projects 
Agency  Information  Processing  Techniques  Office. 

The  research  program  has  as  its  primary  purpose  the  development 
of  techniques  and  systems  for  efficiently  processing,  transmitting,  and  ana- 
lyzing visual  images  and  two  dimensional  data  arrays.  Three  tasks  are  re- 
ported: Image  Understanding  Projects;  Image  Processing  Projects;  Smart 
Sensor  Projects.  The  Image  Understanding  Projects  involve  the  development 
of  generalized  processing  systems  for  analyzing  images  and  extracting  salient 
information.  The  Image  Processing  Projects  include  research  on  novel  im- 
age coding  methods,  image  restoration,  vision  modelling,  and  nonlinear  two 
dimensional  optical  filtering  techniques.  The  Smart  Sensors  Projects  com- 
prise investigations  of  Electronic  and  optical  processing  methods  integrated 
with  imaging  sensors  to  perform  feature  and  symbolic  extraction. 
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1.  Besearch  Project  Overview 

This  report  describes  tae  prograss  sod  results  of  the  University 

} 

of  Southern  California  iaage  understanding  and  inage  processing 
research  study  for  the  period  of  1 Septeaber  1975  to  31  Barch  1976. 

v 

The  research  study  has  been  subdivided  into  three  projects: 

IE 

laage  Understanding  Projects 
Inage  Processing  Projects 
Saart  Sensors  Projects 

The  iaage  understanding  projects  involve  research  directed  toward  the 
goal  of  developing  generalized  processing  systens  capable  of  analyzing 
iaages  and  extracting  salient  inforaation.  Specific  studies  include 
feature  extraction,  syaboiic  description,  interpretation,  and  systeas 
analysis.  The  iaage  processing  projects  include  research  on  novel 
iaage  coding  techniques  based  upon  results  of  the  iaage  understanding 
study,  advanced  iaage  restoration  aethods,  vision  aoielling  in  support 
of  iaage  understanding,  aad  studies  of  nonlinear  two  diaensional 
optical  filtering  techniques  for  the  iapleaentation  of  iaaging 
sensors.  The  saart  sensors  projects  coaprise  investigations  of 
electronic  and  optical  processing  aethods  which  can  oe  integrated  with 
iaaging  sensors  to  perfori  low  level  iaage  enhanceaent  and  feature 
extraction  within  the  sensor.  The  saart  sensors  research  work  is 

| 

being  perforied  by  the  Hughes  Aircraft  Coapany  Besearch  Laboratories 
in  Malibu,  California  under  subcontract  to  USC. 

ii 


Section  2 of  this  report  details  the  research  effort  on  the  iaage 
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under standing  projects.  Taa  inage  processing  research  activities  are 
described  in  Section  3,  and  Section  ft  covers  the  uorli  on  the  snart 
sensors  projects.  Section  5 is  a list  of  publications  by  project 
aenbers  during  the  reporting  period. 
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2.  Image  Understands a g Projects 

The  effort  in  taa  image  understanding  projects  is  directed  toward 
the  eventual  developaant  of  processing  systems  for  generalized  image 
analysis.  Applications  of  such  systeas  include  photointerpretation, 
vehicle  guidance,  visual  tracking,  nap  Batching,  cartography,  and 
iaage  coaaunication . 

The  research  study  is  organized  according  to  tha  structure  of  a 
conceptual  iaage  understanding  systea.  This  systea  consists  of  a 
feature  extraction  stage  which  detects  and  aeasures  priaitive  features 
such  as  edges  and  texture  regions  from  an  input  iaage.  Next  the 
primitive  features  are  grouped  into  aeaningful  symbols  such  as  object 
boundaries,  segaents  of  soae  coaaon  attribute,  or  basic  shape 
structures.  Finally  tae  iaage  syabols  are  interpreted  in  teras  of 
their  seaantic  relationships  to  produce  a concise  quantitative 
description  of  the  original  iaage.  Soae  knowledge  base  is  assuaed 
available  for  guidance  of  all  eleaants  of  the  iaage  understanding 
systea. 

The  philosophic  approach  to  the  study  has  been  to  bring  together 
a research  teaa  ski  Lied  in  digital  signal  processing  and  concepts  of 
artificial  intelligence.  Although  the  research  effort  at  (JSC  involves 
all  stages  of  the  overall  iaage  understanding  systea,  initial  eaphasis 
has  been  deliberately  pLaced  on  the  feature  extraction  and  syabolic 
description  systea  aleients.  Sapid  progress  is  neeled  in  these  areas 
to  assist  the  researcn  in  the  higher  stages  of  the  iaage  understanding 
systea. 


'*,»  1 V ' »IJ| 
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2.1  Lauiiiace  Edge  Detection  Techniques 

Will  id*  K.  Pc  d 1 t 

Changes  ot  discontinuities  ia  an  image  attribute  such  as 
luainance,  tristimulns  value,  or  texture  are  fundamentally  iaportant 
primitive  features  of  aa  image  since  they  often  provide  an  indication 
of  th  '-t  spatial  extant  of  objects  within  the  image.  Local 

discontinuities  in  iaage  luainance  or  amplitude  level  are  called 
luainance  edges,  vhila  global  luainance  discontinuities  are  defined  to 
be  boundaries.  This  report  presents  a survey  of  luainance  edge 

detection  techniques. 


Figure  1 contains  sketches  of  one  and  two  dimensional  luainance 
edges  ouch  represents i as  a raap  increase  in  iaage  amplitude  level 
froa  a Low  to  a high  level.  In  the  one  dimensional  case  the  edge  is 
characterized  by  its  height,  the  slope  angle,  and  x-coordinate  of  the 
slope  aiipoint.  An  edge  exists  if  both  the  slope  angle  and  height  are 
larger  than  specified  critical  values.  For  the  two  dimensional 
example  the  orientation  with  respect  to  the  x-axis  is  also  of 
importance.  An  ideal  edge  detector  processing  the  image  regions  of 
figure  1 should  produce  an  edge  indication  localized  to  a single  pixel 
located  at  the  midpoint  of  the  slope. 

A common  aproacn  to  monochrome  edge  detection  is  illustrated  in 
figure  2 in  which  an  ociginal  monochrome  image  F ( j,  k»  undergoes  a grey 
scale  edge  enhancement  oy  linear  or  nonlinear  processing  to  produce  an 
image  field  G(j,k)  witi  accentuated  spatial  luminance  changes.  Next, 
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a threshold  operation  is  perforaed  to  determine  the  pixel  location  of 
significant  edges,  A negative  going  edge  exists  if 


O0.W<Tti(j,k)  (2.1-1.) 

and  a positive  going  edge  exists  if 

GO.lOfcT'jO'k)  (2.1-lb) 

where  TL(j,k)  and  are  lover  and  upper  threshold  values* 
respectively.  These  threshold  values  nay  be  Bale  spatially  varying  to 
coapensate  for  gross  spatial  lueinance  changes.  Threshold  selection 
is  one  of  the  key  issues  in  edge  detection.  A threshold  level  set  too 
high  will  not  perait  detection  of  low  aaplitnda  structural  iaage 
eleeents.  Conversely*  a threshold  level  set  too  low  will  cause  noise 
to  be  falsely  detected  as  an  iaage  edge.  An  edge  location  nap  E(j*k) 
is  often  generated  to  Indicate  the  positions  of  edges  within  an  iaage. 

For  exaaple*  all  positive  edge  locations  could  be  indicated  by  white 
pixel  values  against  a black  background.  Alternatively*  positive 
going  edges  could  be  noted  by  white  pixels*  negative  going  edges  by 
black  pixels*  and  non-edge  regions  by  aid-grey  pixels. 

A second  najor  approach  to  luainance  edge  detection  involves 
fitting  of  a local  region  of  pixel  values  to  soae  ideal  representation 
of  a one  or  two  diaensional  edge*  as  defined  in  figure  1.  If  the  fit 
is  sufficiently  close*  an  edge  is  said  to  exist*  and  its  assigned 
paraseters  are  those  of  the  ideal  edge. 
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Linear  Edge  BahnsiMnt  Methods:  i variety  of  edge  enhancement 
techniques  can  be  itilized  to  accentuate  eige3  prior  to  threshold 
detection.  One  of  the  simplest  techniques  is  discrete  differencing 
analogous  to  continuous  spatial  diff erentiition.  Horizontal  edge 
sharpening  can  be  obtaiued  by  the  running  difference  operation  which 
produces  an  output  iaage  according  to  the  relation 

C(J,  k)»  F<J,  k)-F(J,  k*l)  (2.  l-2a) 

Similarly,  vertical  sharpening  results  from  the  operation 

<HJ»  k)»  F(J.  k)-F(J+l,  k)  (2.  l-2b) 

Diagonal  sharpening  can  be  obtained  by  subtraction  of  diagonal  pairs 
of  pixels. 

Horizontal  edge  accentuation  can  also  be  accomplished  by  forming 
the  differences  between  the  slopes  of  the  iaage  amplitude  along  a line 
according  to  the  relation 

• G(j,kl  = CF(i,k>-F(j.k-l»ryO*k+D-FO,k)]  (2.  l-3a) 

or  equivalently 

G(j, k)  = 2F(j»k)-F(j.k-l)-FG.k+l)  (2.  l-3b) 

Similar  axpressions  axist  for  vertical  and  diagonal  slope  differences. 

Two  dimensional  liscrete  differentiation  can  be  performed  by 
convolviog  the  original  iaage  array  with  the  compass  gradient  masks 
listed  balow  <1,p.111>: 


North 


1 

l 

-1 


Northeast 


East 


Southeast 


South 


Southwest 


Nest 


1 1 

-2  1 

-1  -1 


I -1 
-2  -1 
1 -1 


(2.  l-4a) 


(2.  l-4b) 


(2.  l-4e) 


(2.  l-4d) 


(2.  l-4e)  j 


(2.  l-4f) 


(2.  l-4g) 


Northwest 


1 

1 

1 


i l i 

1 -2  -1 

I -1  -1 
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(2.  l-4h) 


I 

M 

I 


H = 


The  coapass  naaes  indicate  the  slops  direction  of  aaxiaaa  response, 
e.g.  the  Bast  gradient  a ask  produces  a aaxiaaa  output  for  horizontal 
luainance  changes  froa  left  to  right.  It  should  be  noted  that  the 
gradient  aasks  have  zero  weighting  (the  sua  of  the  array  eleaents  is 
zero)  so  that  there  La  no  output  response  over  constant  luainance 


regions  of  the  isage. 

Edge  sharpening. 

without 

regard  to  edge  direction. 

can  be 

obtained  by  convolution  of  an 

iaage  with  a Laplacian  aask. 

Several 

types  of  Laplacian  aasks 

are  listed  below: 

Mask  1 

0 

-1  0 

H = -1 

4 -1 

(2.  l-5a) 

0 

-1  o 

Mask  2 

-1 

-1  -1 

H = -1 

8 -l 

(2.  l-5b) 

-1 

-1  -I 

Mask  3 

1 

-2  1 

H = -2 

4 -2 

(2.  l-5c) 

1 

-2  1 

Edge  sharpening  sen  be  Bade  proportional  to  the  statistical 
correlation  of  pixel  values  by  the  statistical  sash  <1,p.125> 


I 
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H = 


P„ 

C R 


Vr 


'R(1+,,C  > 


(l+»c2)(l+»l2) 


•'r^'c  » 


PCPR 


-'c<1+fc> 


P P 
C R 


(2.  1-6) 


iu  which  pR  and  Pc  raprosant  the  assuaed  Markovian  correlation  factor 
between  adjacent  row  and  coLuan  pixels.  If  pR*  Pc*0,  there  is  no 
adjacent  eleaent  correlation  and  the  statistical  aask  has  no  effect; 


in  the  extreae,  if  PR  =P  the  statistical  aask  reduces  to  the 


Laplacian  aask  of  eg.  (5c)  . 


Argyle  <2>  and  Hacleod  C3#4>  hare  proposed  Gaussian  shaped 
weighting  functions  as  a aeans  of  edga  enhancaaent.  The  Argyle 
function  is  a split  Gaussian  function  defined  in  one  diaension  as 


h(x)  = exp 


t# 


xiO 


b(x)  » -exp  M£|)  x < 0 

where  p is  a spread  constant.  The  dacleod  function  given  by 


K;fi 


(2.  1-7) 


Hfe,y)  » exp 


{*(♦)  } }] 


(2. 1-8) 


where  p and  t are  spread  constants.  This  function  suppresses  the 
effect  of  pixel  values  in  the  edge  transition  region  and  edges  in  rows 
above  and  below  the  edge  to  be  detected. 


A cosaon  Unit  at  ion  of  the  linear  edge  sharpening  aethods 
previously  discussed  is  the  aaplif ication  of  high  spatial  frequency 
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noise  and  artifacts  is  a result  of  the  inheraat  differencing 
operations  involved.  Noise  smoothing  cm  ba  encorporated  into  the 
linear  edge  sharpening  aethods  by  performing  tne  linear  sashing  on 
regions  of  pixels  citner  than  on  individual  pixels  <5>.  This  can  be 
accomplished  by  forming  a linear  mask 

H(J,k)  » Hg(j,k)  • H(j,k)  0 

by  convolving  one  of  taa  edge  anhanceaant  masks  previously  defined 
with  a low  pass  filter  averaging  mask  Hs(j,k).  Such  spatial 
averaging*  of  course,  laads  to  a smoothing  of  edges  as  veil  as  noise. 


Nonlinear  Edge  Bnaancement  Methods:  Nonlinear  edge  detection 

systems  parform  nonlinear  combinations  of  pixel  values  as  a means  of 
edge  enhancement  prior  to  thresholding.  Rost  technigues  are  limited 
to  processing  over  small  2 x 2 or  3 x 3 pixel  uindovs. 

Roberts  <6>  has  introduced  the  nonlinear  cross-operition 

GR(jfk)=^r(j,k)-r(j+I,k+|,]2+[F(jfk+1)-F(j+1,k)]2)l/2  (: 

as  a tvo-diaensional  differencing  method  for  edge  sharpening  and  edge 
isolation.  Another  spatial  differencing  operation*  nhich  is  of 
computationally  simpler  form*  im  givmm  by 

GAU*k)=lr(j.k)-F(j+l.k+l)|+|F(j,k+l)-F(j+l,k)|  ( 


It  can  be  easily  shoma  that 


oR(j,k)*  oA(j.k|*  /TO_(j,k) 


(2.1-12) 
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Crude  directional  inebriation  can  be  extracted  ay  noting  which  of  the 
four  pixels  is  largest  at  a detected  edge  point. 

Sobel  <7,p.271>  sis  suggested  a 3 x 3 nonlinear  edge  enhanceaent 
operator  described  by  the  pixel  nuabering  convention  of  figure  3.  The 
edge  enhanceaent  plana  is  defined  as 

G(J.k)  Y2  (2.1-13*) 

* « 

where 

X=  (A2  + 2A3+A4)  - (Aq+2A7+A6)  (2.1- 13b) 


e i 


Y = (AQ+  2Aj+A2)  - (A6+2A5+  A4) 


(2.  l-13c) 


Another  3x3  nonlinear  edge  enhanceaent  algoritha  has  been 
introduced  by  Kirsca  <3>.  Referring  to  the  notation  of  figure  3,  the 
enhanceaent  is  given  as 

7 


G(j,k)  = xnax  |l,  max  [ | 5St-  3T.  | J | 


(2.  1- 14a) 


where 


h ~ Ai+Al+1+  Ai+2 


Ti"  Ai+3+  Ai+4+  Ai+5+Ai+6+Ai+7 


(2.  l-14b) 


(2.  l-14c) 


The  subscripts  of  S and  T are  evaluated  aodulo  8.  Basically,  the 
Kirsch  operator  provides  the  aaxiaal  coapass  gradient  aagnitude  about 
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an  inage  paint  ignoring  the  pixel  value  P(j,k|. 

Hall is  <9>  has  proposed  a Qonlinear  edge  detection  schene  based 
upon  hononorphic  inage  processing.  According  to  this  schene  an  edge 
exists  if  the  aagnituia  of  the  logaritha  of  the  iaage  luainance  at  a 
pixel  exceeds  the  aagaitude  of  the  average  logarithaic  luainance  of 
its  four  nearest  neighbors  by  a fixed  threshold  value,  iith  reference 
to  figure  3,  the  edge  euhanceaent  plane  is  defined  as 


G(j,k)  * logtF(j,k)]-  * lof  (A1)-ilog(A3).ilog(A5)-ilog(A7)  (2.  l-15a) 
or 


G(j,k)  =ilog  I— ISsi! 1 (2.1.I5b) 

IVsVtJ 

Comparison  of  G(j,k)  against  upper  and  lower  threshold  values  is 
exactly  eguivalent  to  coaparison  of  the  fraction  in  the  brackets  of 
eg.  (15b)  against  a aodified  threshold.  Therefore,  logarithas  need  not 
be  coaputed.  The  principal  advantage  of  the  logarithaic  edge  detector 
besides  its  coaputational  simplicity  is  that  the  technigue  is 
insensitive  to  aultiplicative  changes  in  luainance  level. 


The  logarithaic  edge  enhanceaent  scheae  defined  by  eg.  (15)  can  be 
considered  as  a linear  enhanceaent  with  the  Laplacian  aask  of  eg. (5a) 
perforaed  on  the  logarithas  of  the  pixel  values.  In  this  context 
other  edge  enhanceaent  aethods  can  be  easily  foraulated  as  a 
concatenation  of  point  nonlinear  operations  followed  by  linear  edge 
enhanceaent  and  thresnolding. 


j 


ft 
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Rosenfeld  <10>  has  developed  a nonlinear  product  averaging  east 
for  edge  sharpening  and  edge  isolation.  Hith  this  operator  the 
running  one-diaensional  average 

DM(j,k)»  -jj  [F{j+M-l.k)+r(j+M-2,k)+. ..+F(j,k) 

-F(j-l,k)-F(j-2,k)-...F(j-M,k)]  (2.1-16) 

is  forael  at  each  pizal  point  where  i*2mand  a is  aa  integer.  This 
operation  is  perforael  for  8«1 ,2,4, 8,16, . .. ,atc.  up  to  soae  desired 
upper  liait.  Then,  tae  product 

J*  k)*D  j(j,  kJD^O,  k).  . . D^j(j,  k)  (2. 1-17) 

is  forael  at  each  pixsl.  Conceptually,  the  higher  order  averaging 
aasks  provide  a broad  indication  of  edges  anl  soae  degree  of  noise 
suppression,  while  taa  Lover  order  averaging  aasks  yield  localized 
derivatives,  which  ars  such  aore  noise  sensitive.  Together,  it  is 
hypothesized,  the  product  of  the  variable  length  averaging  aasks 
should  give  a positive  indication  in  the  vicinity  of  true  edges. 
Rosenfeld 's  justification  of  the  operation  is  as  follows:  N...the 

result  ^mUA)  tenl3  to  yield  sharply  localized  detections  of  aajor 
edges  while  suppressing  noise.  Intuitively,  this  is  because  the 
product  is  large  only  when  all  factors  are  large,  and  as  soon  as  one 
novas  away  froa  a position  "just  at"  and  edge  point,  the  factors  with 
low  a's  becoae  snail;  waile  if  one  is  not  at  or  near  a aajor  edge,  the 
factors  with  high  a's  are  snail. " 

Rosenfeld  <11>  has  also  proposed  a nonlinear  processing  procedure 
for  isolating  large  sharp  edges  in  the  neighborhood  of  snaller  edges. 
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This  procedure,  which  will  be  called  dominant  neighbor  suppression,  is 
perforaed  by  scanning  the  edge  enhanced  pline  G(j,k|  with  a seall 
pixel  window.  The  valua  of  G(j,k)  in  the  center  of  the  window  is 
suppressed  (set  to  zero)  unless  its  aagnitule  is  tae  greatest  of  all 
saaples  within  the  window.  Conventional  aaplitude  thresholding  then 
follows.  A variation  of  the  process  is  to  perait  suppression  of 
3(j,k)  only  if  a neigabor  in  the  window  doainates  by  a significant 
aaount.  The  doainant  neighbor  suppression  thresholding  algoritha  has 
proven  guite  effective  for  edge  detection  when  coupled  with  an  edge 
enhanceaent  aethod  that  provides  soaa  noise  saoothing. 


Edge  Pitting  Hetaols:  Ideal  edges  nay  be  viewed  as  one  or  two 
diaensional  raap  signals  of  the  fora  sketched  in  figure  1.  Actual 
iaage  data  can  then  be  aatched  against,  or  fit  to,  the  ideal  edge 
aodels.  If  the  fit  is  sufficiently  accurate  at  a given  iaage 
location,  an  edge  is  a35uaed  to  exist  with  the  saae  paraaeters  as  the 
ideal  edge  aodel. 

In  the  one  diaensional  edge  fitting  case  described  in  figure  4 
the  iaage  signal  f (x)  is  fit  to  a step  function 


x < x( 
x * 0 


(2.  1-18) 


An  edge  is  assuaed  present  if  the  Bean  square  error 


-/ 


VL 


[f(x)-a(x)]  dx 


VL 


(2.  1-19) 
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is  below  soae  threshold  value.  In  the  two  dimensional  formulation  the 
ideal  step  edge  is  defined  as 


S(x,y)  = 


b x comO-h  y win  0 < p 

bfh  x cow0+  y win  0 i p 


where  0 and  P represent  the  polar  distance  from  the  center  of  a 
circular  test  region  to  the  normal  point  of  the  edge.  The  edge 


fitting  error  is 


l=f f [F{x,y) 


S(x*  y)3  dxdy 


circle 


Hueckel  <12>  has  developed  a procedure  for  two  dimensional  edge 
fitting  in  which  tne  image  points  within  the  circle  of  figure  4 are 
expanded  in  a set  of  two  dimensional  basis  functions  by  a Fourier 
series  in  polar  coordinates.  Let  H.  (x, y)  represent  the  basis 
functions.  Then  the  weighting  coefficients  for  the  axpansions  of  the 
image  and  the  ideal  step  edge  become 


■•■//  Hi(x,yJF(x#y)dxdy 
Hi(x,y)S(x,y)dxdy 


It  should  be  noted  that  S(x,y)  is  defined  parametrically  in  terms  of 
the  set  (b,h,P,8).  In  Hueckel's  algorithm  the  expansion  is  truncated 
to  eight  terns  for  computational  economy  and  to  provide  some  inherent 
noise  smoothing.  Minimization  of  the  mean  square  difference  of 
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•4.(21)  is  equivalent  to  ainiuization  of  foe  all 
coefficients.  Huec&al  has  parf oread  this  niniaization,  invoking  soae 
siapliffing  approxina tions,  and  has  foraulatad  a set  of  nonlinear 
agnations  expressing  tie  edge  parameter  set  (b,h,p,9)  in  terns  of  the 
•xpaasioas  coefficients  ai . A test  is  perforaed  to  coapare  the  actual 
iaago  data  P (x,y)  vith  the  edge  fit*  and  if  thi  resulting  fit  is  poor 
becaaso  of  noise,  no  edge  is  judged  present.  If  the  fit  is 
sufficiently  close,  a farther  test  is  Bade  to  de tannine  if  the  edge 
contrast  factor  h is  j ranter  than  a threshold  factor. 

The  coaplexity  of  the  Hueckal  algoritha  renders  it  difficult  to 
analyze  theoretically.  However,  experiaental  evidence  indicates  that 
the  Haeckel  operator  is  guite  sensitive  to  the  detection  of  low 
contrast  edges  in  reasonably  noisy  iaages. 
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2.2  A New  Class  of  Bdga  and  Feature  Detection  Operators 
Herner  Frei  and  Chung-Ching  Chen 

Hunan  or  aachine  scene  analysis  depends  greatly  upon  the  ability 

to  partition  an  inaga  into  regions  that  correspond  to  individual 

% 

objects.  One  popular  approach  is  the  detection  of  edges,  which  are 
crudely  defined  as  the  boundaries  between  isage  regions  having 
different  properties.  Such  properties  are  brightness,  color,  texture 
or  whatevar  night  enania  the  detection  of  object  boundaries  within  the 
inage.  Clearly  then,  edge  detection  oust  be  based  upon  the  inspection 
of  sore  than  one  picture  elenant  within  sub-areas  of  an  inage  in  order 
to  decide  whether  an  adge  segnent  is  present  within  each  area.  This 
operation  can  be  expressed  as 

edge  = 1tf{A))  (2.2-1) 

where  f (•)  represents  sone  function  of  the  picture  elenents  within 
area  A.  and  T(»)  is  a threshold  or  decision  f auction.  The  resulting 
edge  elenents  can  be  characterised  by  variables  such  as  aaplitude. 
orientation,  position,  and  a confidence  factor. 

Upon  exaninatioQ  of  the  whole  picture,  object  boundaries  are 
constructed  by  connacting  the  edge  elenents  detected  into  lines,  an 
operation  that  can  ba  based  upon  siaple  syntactic  rules,  for  exanple. 
to  connect  neighbor  adge  elenents  that  have  sinilar  orientation  and 
delete  isolated  elenents.  Sone  of  the  difficulties  of  edge  detection 
are  created  by  noi3a.  but  such  nore  so  by  the  facts  that  visually 
distinct  edges  sonatinas  cannot  be  detected  within  an  area  of  size  A. 


Page  23 


or  conversely,  that  mat  appears  to  be  an  edge  sithin  A soaetlees 
belongs  to  a hoaogenaoasly  textured  area  of  the  picture.  Increasing 
the  sise  of  A would  apparently  solve  the  problenf  except  for 
cosputational  lini tattoos.  Also  note  that  if  the  size  of  A is 
increase!  greatly,  tha  description  of  an  edge  within  A becoaes  very 
coeplex.  It  is  generally  recognized  that  boundary  detection  is 
therefore  best  dona  oy  the  cosbination  of  a relatively  siaple  edge 
operator  followed  by  algorithms  such  as  thinning  and  linking  that 
delete  irrelevant  alga  points  and  connect  the  reaaining  ones  into 
lines.  A nunber  of  suca  edge  detection  operators  have  been  published, 
for  exaeple,  Roberts  gradient  <1>,  Kirsch  <2>,  Sobel  <3>,  Prewitt  <«>, 
saoothed  gradient  <5>,  Robinson  <6>  and  Hueckel  <7,3>.  All  of  the 
above  algorithns  can  oe  expressed  in  general  taras  as 

"edge"  * T {f(  3 *A)}  (2.  2-2) 

where  A is  a linear  tcaasf oraation  of  A and  f is  a non-linear  distance 
function.  Edge  detection  is  then  accosplishad  by  sowing  the  area  A 
over  the  whole  picture  3eguentially,  and  recording  the  edge  paraseters 
found.  Hithin  that  class  of  algorithns,  the  cosputationally  fast 
operators  consist  of  a 3et  of  templates  R of  size  2x2  or  3x3 
pixels  that  are  coavolved  simultaneously  with  the  image  to  be 
analyzed.  The  saxinua,  sum  of  sguares,  or  absolute  sum  of  the 
convolutions  is  then  thresholded  to  obtain  edge  points.  Edge 
direction  can  also  be  determined  in  a straightforward  manner,  either 
by  identifying  the  orientation  of  the  template  which  yields  the 
largest  convolution  value  at  each  point,  or  by  an  arctangent 


Page  24 


[ 

coaputatioo  <1-6>. 

Coop&rison  of  the  above  operators  reveals  sLsilarities  that  can 
be  attributed  to  the  touristic  design  approaches  used.  let  there  does 
not  sees  to  exist  a generally  recognised  criterion  for  optiaization; 
the  evaluation  of  such  an  algorithn  is  perforsed  chiefly  by  visual 
inspection  of  processed  edge  pictures.  The  exception  is  the  Hueckel 
operator,  in  uhich  aa  ideal  edge  is  defined  a priori,  and  the 
algorithn  optinised  to  detect  snch  ideal  edges. 

The  present  contribution  follows  a sinilar  approach  in  the  sense 
that  ideal  edge  elenaat3  are  defined  first.  The  analysis  reveals  that 
no  nore  than  two  tenpiates  are  required  for  the  optiaal  detection  of 
such  edges,  and  the  paraaeters  of  the  operator  can  be  adjusted  to 
natch  preferred  types  of  non>ideal  edges  (sharp  versus  saooth) . Given 
the  axionatic  criterion  of  ideal  edge  elenents,  nost  existing 
algorithns  can  be  readily  evaluated,  and  it  is  easy  to  verify  that 
aany  are  coaputationaLlf  redundant. 

The  edge  operator  is  then  generalized  for  other  inage  features 
such  as  ideal  line  eleaents  and  discrete  Laplacian  (ideal  points) 
which,  taken  together,  constitute  a rudiaentary  but  fast  orthogonal 
feature  transforn.  Exaaination  of  the  characteristics  of  the 
transfora  dona in  suggests  the  elaboration  of  a aeasure  of  confidence, 

• I 

or  conversely,  the  design  of  adaptive  threshold  strategies  for  edge 
and  line  detection.  Pucthersore,  experinental  results  suggest  that 
the  above  features  provide  appropriate  elenents  for  contextual 
thinning  and  linking  algorithns. 
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Definitions:  Consider  an  image  region  k of  size  n x n sampling 
intervals  shown  in  figure  la.  An  ideal  edge  eleaent  in  the  continuous 
isage  doaain  is  defined  as  a straight  boundary  passing  through  the 
center  of  A that  separates  two  regions  of  different  average  brightness 
bj  and  b2.  Adopting  the  convention  "bj  > t>2 , the  direction  $E  of  the 
edge  eleaent  is  unigaely  deterained  with  respect  to  any  arbitrary 
direction  (figure  1b|  . Hote  that  the  ideal  edge  element  is 
characterized  by  its  aagnitude  lb1-b2l  and  orientation  $ E#  where 
0 <_$E  <_  2tt.  Possible  discrete  representations  of  the  sampled  ideal 
edge  eleaent  will  now  be  examined. 


of 


2 

Consider  the  set  of  n brightness  samples  b..  of  A as 
2 

an  n -dimensional  vector  space  B . The  elements 


an  element 
of  B can  be 


represented  by  a matrix  B or  a column  vector  b. 


For  example  for  n = 3 


(2.  2-3a) 


(2.  2- 3b) 


An  inner  product  (dot  product)  on  B is  defined  is 
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n n 

(S,£)  = S E v«  (2.2-4.) 

U1  j=l 

2 

a 

=iCbiCi  (2. 2- 4b) 

i=  1 


All  of  the  fast  edge  operators  aentioned  previously  can  non  be  easily 
expressed  in  ter.s  of  the  above  notation.  For  exanple  Roberts 
gradient  is  given  by 


edge  = T((B,  Yfj)2  + (BuW2)2] 

with 

wl*h  *1  ■ [i  -d  *2  -{?  o] 


(2.2-5) 


(2.  2-6) 


where  T (• ) * 0 or  1 if  the  argusent  is  ssaller  or  larger, 

respectively,  than  a preset  value.  Mote  that  J(1  and  J J.2  can  be 
regarded  as  a pair  of  basis  vectors  for#  . The  following  hypothesis 
is  now  fornulated: 


The  discrete  representation  of  all  ideal  edge  elesents  defined 
above  foras  a proper  subset  6 of  B , of  diaension  two.  Is  a 
consequence,  there  aust  exist  a basis  consisting  of  Jlj  and  Jl2 
which  spans  <?  such  that  all  ideal  edge  eleaents  can  be 
represented  by  no  sore  than  two  linearly  independent  coefficients 
as 


*t=  (5,^) 


(2.  2-7) 


f 
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for  i * 1,2. 

Note  that  this  foreulation  is  intuitively  satisfying  in  that  ideal 
edge  elenents  are  characterized  by  teo  independent  quantities,  their 
nagnitade  and  direction.  If  and  J2  ara  orthonornal,  these 

quantities  can  be  easily  obtained  froe  j£j  and  X2  bF 

"magnitude"  * (f^  ♦ (2. 2- 8a) 

"orientation"  = atan  ill 

i sore  satisfying  proof  can  be  obtained  by  observing  that  all  possible 

e 

discrete  representations  of  non-ideal  edge  alessnts  can  be  obtained  as 
a linear  conbination  of  JLj , JL2  and  other  non-edge  basis  vectors  to  be 
discussed.  It  has  been  found  however  that  for  n > 2,  the  subset d is 
not  unique,  but  depenls  upon  the  veighting  function  used  to  coapute  Jtj 
and  Ji2«  Figure  2 sms  ezaaples  of  such  basis  vectors  for  n » 2 and 
n ■ 3.  Mote  that  uhat  distinguishes  the  bases  of  figures  2b,  2c  and 
2d  are  the  weights  given  to  pizels  as  a function  of  distance  froa  the 
separating  boundary.  This  peraits  tuning  of  tha  edge  operator  to  a 
certain  eztent  for  the  detection  of  sharp  or  saooth  edges.  It  is  now 
possible  to  assess  eristing  edge  operators  by  projection  of  their 
teaplates  onto  the  basis  vectors  found.  One  conclusion  that  can  be 
draws  laaediately  is  that  any  operator  that  ashes  use  of  sore  than  two 
teaplates  is  cosputatiosally  redundant  because  tha  excess  teaplates 
are  linear  conbinatioas  of  two  edge  basis  vectors.  Consequently,  the 
excess  convolutions  are  linearly  dependent  as  well,  and  therefore 
redundant.  Figure  3 saows  projections  of  the  tanplates  of  the  Kirsch 


) 


(2.  2-8b) 


L 


a)  n = 2; 


two  possible  bases  for  £ 


i 


b ) n =3 


c ) n = 3 


d ) n-3 


Figure  2.  2-2.  Possible  orthogonal  basis  vectors  spanning  £for  n=2  and  n=3. 

(The  factors  to  the  left  of  the  templates  render  the  basis  vectors 
orthonorrnal.  Often  not  necessary). 
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operator  onto  the  basis  vectors  of  figure  2.  i sore  detailed  stud;  of 
existing  operators  cai  he  Bade  Later,  b;  observing  that  the  teaplates 
used  are  soaetiaes  a Linear  coabination  of  Bj , and  other  non-edge 


basis  vectors  to  be  liscussed.  Figure  4 shows  two  convolutions  of  a 
test  iaage  (figure  4a)  with  the  basis  vectors  Bj  and  fj2  of  figures  4b 
and  4c.  Edge  detection  could  be  done  at  this  stage  b;  thresholding 
the  nagnitude  of  the  projection  of  _B.  onto  B shown  in  figure  4d. 

Other  Features:  Tha  edge  detection  operator  can  now  be  extended 
to  the  detection  of  other  iaage  features  of  interest.  An  ideal  line 
eleaent  is  defined  in  tie  continuous  iaage  doaain  as  a straight  strip 
of  width  apprcxiaateiy  agual  to  one  saapling  interval,  passing  through 
the  center  of  A,  and  of  different  average  brightness  "b1  than  its 
surround  "b2  (figure  1c).  The  ideal  line  elassnt  is  characterized  b; 
its  direction  $£,  its  aagnituda  and  its  color  defined  as 
sgnfbpb^).  The  discrete  representation  of  all  ideal  line  eleaents 
fores  another  subset  £ otB  , of  diaension  two,  which  shares  only  the 
zero  eleaent  with  <S  . An  intuitive  set  of  tasplates  fora?  shown  in 
figure  5a  can  be  reduce!  to  two  line  basis  vectors  J[3  and  and  two 
point  basis  vectors  and  Since  the  latter  do  not  have  a 
directional  property,  and  since  they  span  the  space  of  all  possible 
discrete  realizations  of  the  Laplacian  operator  <9>,  it  will  be 
assuaed  that  Jl5  and  J L6  span  an  independent  point  subset,  whereas  the 
line  subspace  of  s£  is  spanned  by  Jl3  and  It  should  be  noted  that 
there  exists  an  aabiguity  in  as  to  the  color  of  Lines,  that  can  be 
resolved  by  exaaination  of  the  coaponents  in  tha  point  subspace. 


A- 


(c)  Convolution  with  W 
(scaled) 


(d)  Magnitude  of  projection 
of  original  image  onto 
the  edge  subspace  g 


Figure  2.2-4.  Original  image,  projections  onto  the  basis  vectors  of 

the  edge  subspace,  and  magnitude  of  projection  onto  the 
same  subspace. 
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Figure  2.2-5.  "Intuitive"  line  detection  templates  and  decomposition 
into  line  and  point  basis  vectors. 
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The  basis  can  ba  extended  to  span  the  entice  space®  . The  last 
three  basis  vectors  found  are  M0  (average  brightness)  and  JL7,  JJg  which 
span  a fifth  subset,  tentatively  called  a ridge  subset  which  could  be 
seaningful  in  texture  analysis.  figure  6 shows  one  coaplete 
orthonorsal  basis  for  a * 3,  expanded  froa  the  edge  basis  vectors  of 
figure  2b.  Figure  7 shows  the  convolutions  of  the  inage  with  each  one 
of  these  vectors  (see  also  figure  4b  and  4c). 

It  is  now  possible  to  define  a feature  vector  X of  diaension 

2 

saaller  or  egual  to  n , whose  coaponents  are  the  projections  (l-,fi)  of 
JL  onto  the  respective  basis  vectors  above.  For  exaaple,  for  n * 2 in 
the  expansion  of  the  basis  of  Boberts  gradient 


dote  that  in  this  case  there  exists  no  poiat  subspace  in  B , and  the 
line  subspace  has  only  one  diaension.  Therefore,  the  direction  and 
color  of  ideal  lines  cannot  be  resolved  when  n = 2. 

In  order  to  increase  coaputational  spead,  the  aagnitude 
is  soaetiaes  replaced  by  the  sun  of  absolute  values 

"magnitude"  = |fj|  + |f2| 


function 


(2.2-10) 
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(2.2-9) 


J 


which  can  be  in  error  by  up  to  4 IX.  It  has  been  suggested  that  such 
errcrs  do  not  seriously  affect  the  perforaance  of  edge  detectors,  but 
it  is  easy  to  see  that  they  give  preference  to  certain  orientations. 


Figure  2.2-6.  One  possible  orthogonal  basis  for  B (orthonormal  if  the  factors  to  the 
left  of  the  basis  vectors  are  included). 


Convolutions  of  original  image  with  basis  vectors 
w_.  W..  W_.  W, . W„.  W0  (W,  and  See  figu 
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Extensions:  Several  properties  of  the  feature  transform  are  now 
being  actively  explore!.  Of  particular  interest  are  the  eagnitudes  of 
the  projections  of  JB  onto  the  subspaces<?  ,s£  ,9  ,R  . Figures  4d  and 
8a,  8b,  8c  show  these  eagnitudes  conputed  as 


itude"  = (fj  + f?  )* 


"magnitude1 


for  (i,  j)  = (1,2)  , (3, 4|  , (5,6),  (7,8).  Since  these  eagnitudes  are 
indicative  of  the  respective  contributions  of  ideal  edge,  point  and 
ridge  eleeents  present  within  an  actual  ieage  segeent,  it  appears  that 
a neasure  of  confidence  can  be  derived  as  a function  that  coapares  the 
eagnitudes  of  the  projections  of  A onto  the  edge  subspace  <$  and  the 
non-edge  subspaces  s£  ,9  , ft  , respectively.  Conversely,  it  is 
conceivable  that  the  threshold  decision  can  be  based  upon  the  relative 
eagnitudes  of  the  coiponents,  instead  of  using  a fixed  threshold 
value.  This  is  illustrated  conceptually  in  figure  Id,  where  the 
vector  would  be  classified  as  an  edge  but  not  vector  although 
Jj  £ has  a larger  projection  onto  the  edge  subspace,  and  would  be 
detected  as  an  edge  by  conventional  enhanceaent/threshold  edge 
detection  algorithas. 


Next  it  is  observe!  that  the  nagnitude  inages  of  figures  4a,  8a, 
8b,  and  3c  reveal  a coapleaentary  nature  of  the  edge  and,  to  a lesser 
extent,  ridge  versus  line,  and  point  subspaces.  An  intuitive  arguaent 
for  this  observation  is  aade  by  considering  a non-ideal  line  of  width 
larger  than  one  saapling  interval.  It  is  easy  to  see  that  edge  points 


1 


Figure  2. 2-8. 


Magnitudes  of  projection 
the  three  subspaces  L, 


s of  original  image  (fig.  4a)  onto 
P,  R resp.  (see  also  figure  4d) 
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will  bo  detected  oa  each  side  of  the  line.  Conversely,  lines  and 
points  ate  detected  on  each  side  of  non-ideal  alges,  a fact  that  is 
less  apparent  intuitively,  but  can  be  verified  readily  in  the  pictures 
of  figures  4d , 8a,  80,  and  8c,  or  by  aanual  computation.  This 
observation  suggests  that  contextual  information  supplied  by  the 
regaining  elements  of  tae  feature  vector  ,f  can  be  used  to  eliainate 
edges  due  to  noise,  as  well  as  to  thin  and  lint  edges  or  lines,  since 
any  edge  point  is  surrounded  by  line  points  of  siailar  orientation,  as 
well  as  point  eleaents,  and  vice-versa. 

Conclusions;  It  t3  believed  that  the  present  feature  transfora  is 
a powerful  tool  for  iaage  analysis,  and  enables  an  objective 
analytical  evaluation  of  edge  detectors.  In  addition  to  the 
extensions  aentioned  above,  it  is  also  pointed  out  that  the  present 
approach  can  bu  extended  iaaediately  for  n > 3.  However,  it  appears 
that  arbitrary  large  values  of  n are  not  very  desirable  in  terns  of 
coaputational  load,  because  the  contextual  technigues  outlined  could 
give  siailar  results  at  a lower  cost. 
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2.3  Detection  and  ColLag  of  Edges  using  Directional  Basks 
Guner  S.  Robinson 

A new  inage  coliag  systea  is  described,  which  coabines  the 
detection  and  coding  of  visually  significiant  elges  in  natural  isages. 


I 
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Edg»s  are  defined  as  aaplitude  discontinuities  between  different 
regions  of  an  iaaga.  The  edge  detection  systea  antes  use  of  3 x 3 
easts,  which  are  well  suited  for  digital  iapleaenta tion.  Edge  angles 
are  quantized  to  eigat  equally  spaced  directions,  suitable  for  chain 
coding  of  contours.  Usa  of  an  edge  direction  nap  iaproves  the  siaple 


thresholding  of  gradient  aodulus  iaages.  The  concept  of  local 
connectivity  of  the  edge  direction  aap  is  useful  in  inproving  the 
perforaance  of  edge  operators  such  as  Kirsch  and  Sobal  operators.  The 
concepts  of  an  "edge  activity  index"  and  a "locally  adaptive 
threshold"  iaprove  perforaance  even  further. 

Directional  Hast  Operators;  Two  diaensional  aast  operators  are 
coaaonly  used  for  filtering  and  enhanceaent  <1,2>.  Two  diaensional 
discrete  differentiation  can  be  perforsed  by  convolving  the  original 
iaage  with  the  coapass  gradient  aasts  shown  in  figure  1.  The  coapass 
naaes  indicate  the  slops  direction  of  aaxiaua  response,  e.g.  the 
North  gradient  aast  produces  a aaxiaua  output  for  vertical  luainance 
changes,  i.e.  for  horizontal  edges.  The  direction  of  this  horizontal 
edge  could  be  froa  Left  to  right  or  froa  right  to  left.  In  the 
following,  edge  dir actions  corresponding  to  the  eight  coapass 
gradients  are  detecainad  such  that  the  bright  side  of  the  edge  is 

I 

always  to  the  left  as  one  aoves  in  the  direction  of  the  edge.  The 
diractions  of  the  adgas  corresponding  to  the  eight  coapass  gradient 
Basks  are  also  shown  in  figure  1.  The  nunbers  3,1,... .,7  are  used  for 


the  eight  principal  directions  in  a 3 z 3 grid,  as  shown  in  figure  2. 
This  notation  is  used  ia  order  to  be  consistent  with  the  Preenan  chain 


Direction  of  Prewitt  Kirscn  Three-level  Five- level 

Direction  of  Edge  Gradient  Masks  Masks  Simple  Masks  Simple  Masks 


Figure  2.  3-1.  Examples  of  Compass  Gradient  Masks 


coding  schene  <3,  p.249>  for  subsequent  chiin  coding  of  extracted 

boundaries. 

The  Kirsch  open  tor  <18>  can  be  thought  as  taking  the  output  of 
one  of  the  eight  coapa3s  gradient  nasks  shown  in  tha  second  coluan  of 
figure  1.  The  aask  waich  produces  the  naxinua  output  detersines  the 
direction  of  the  edge.  The  gradient  iaaga  is  obtained  by  taking  the 
aagnitude  of  the  output  of  that  aask. 

A siaple  set  of  coapass  gradient  nasks  can  be  foraed  by  rotating 
the  differentiation  lasts 
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given  by  Prewitt  <1,p. 138)  , as  shown  in  tha  third  coluan  of  figure  1. 

These  directional  lists  are  called  three* level  siaple  nasks  in  order 

to  distinguish  then  fron  the  five-level  directional  nasks  which  will 

be  given  later.  Taa  two  orthogonal  aask3,  B and  B . which  neasure 

x y 

the  gradients  in  the  (forth  and  Best  directions,  approxiaate  the 
partial  derivatives  in  the  x-direction  and  y-dlrection,  respectively. 

Application  of  B and  B to  an  inage  results  in  spatial 

x y 

differentiation  in  two  orthogonal  directions.  The  gradient  aagnitude 
and  direction  can  tain  be  obtained  by  taking  the  aagnitude  and 
direction  cosines  at  each  point.  An  analog  gradient  picture  results 
when  the  gradient  aagaitudas  are  displayed  as  gray  values. 
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The  Sobel  operator  <5,p.271)  can  be  thought  as  a combination  of 
two  graliant  easts,  >ne  in  the  North  and  the  other  in  the  East 
direction.  The  outputs  of  these  two  easts  approxiaate  the  partial 
derivative  in  their  respective  directions.  I gradient  modulus  ieage 
is  obtained  by  tating  the  eagnitude  of  the  two  orthogonal  east 
outputs.  The  direction  of  the  edge  can  also  be  deterained  from  the 
outputs  of  these  two  easts. 

In  the  following  a set  of  five-level  simple  directional  easts  are 
considered  for  obtaining  the  analog  gradient  ieage  and  the  edge 
direction  in  a simple  Banner.*  These  easts,  called  five-level  sieple 
easts,  assuae  five  integer  weights  between  -2  and  *2.  The  five-level 
sieple  coapass  gradient  easts  are  shown  in  tha  fourth  colunn  of  figure 
1.  The  advantages  of  tne  particular  choice  of  weights  are  that: 

la)  The  two  orthogonal  easts 
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approximate  tha  partial  derivatives  in  the  x-direction  and 
y-direction,  respectively. 

(b)  The  zero  weights  in  the  center  of  the  easts  result  in 
ignoring  the  transient  line  where  an  edge  eight  occur. 

(c)  Computing  tha  output  of  the  first  four  easts  is  enough  to 
obtain  both  the  analog  gradient  ieage  and  the  edge  direction  cap. 
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For  exaaple,  if  the  result  of  the  North  east  is  positive,  then 
the  edge  direction  is  "0"  md  if  it  is  negative,  then  the 
direction  is  "4”. 

(d)  i defocusing  operation  can  be  perfornei  on  an  iaage  by  using 
a low-pass  east  such  as 


1 2 1 

M0=  T?  2 4 2 

1 2 1 


which  has  integer  weights  siailar  to  H and  H . This  type  of 

x y 

defocusing  east  could  be.  used  for  obtaining  local  threshold 
values  in  edge  iataction  by  siaple  directional  Basks.  Mask  MQ 
has  a better  sidelobe  structure  than  the  low-pass  counterpart 

of  H and  H 

x y 


W0=?  1 1 1 

ll  1 1 


1 1 


which  nakes  a siaple  averaging  operation  of  niae  pixels  in  a 

3x3  grid  of  aa  iaage.  It  has  been  verified  that  Hn,  H and  M 

0 x y 

are  nenbers  of  a saven-level  orthogonal  set  of  nasks  siailar  to 
the  set  given  by  Frei  and  Chen  <6>. 


(e)  Five-level  siaple  directional  nasks  yield  a higher  gradient 
aaplitude  in  tua  diagonal  direction  than  the  horizontal  or 

I 

vertical  direction.  It  is  known  that  the  visual  acuity  in  the 
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horizontal  and  vartical  directions  is  superior  to  the  visual 
acuity  in  the  diagonal  directions  for  a large  range  of  spatial 
frequencies,  aal  it  is  conjectured  that  the  basis  of  this 
phenonenon  lies  in  the  retina  and  higher  order  visual  pathways 
<7,8>.  The  directional  masks  of  figure  1 can  be  compared  in  the 

* g 

x-y  domain  definal  by  the  partial  derivative  operators  W and  H . 

x y 

For  example,  the  direction  and  normalized  amplitude  of  the  edge 
corresponding  to  tie  Northwest  diagonal  mask  is  described  below: 


Type  of  Mask 

Prewitt 

Kirsch 

Three-level  simple 
Five-level  simple 


Le  (Degrees) 


Amplitude 

0.  943 
0.943 
0.943 
1.067 


This  table  shows  taat  all  four  types  of  directional  masks  are 
nonisotropic.  However  only  the  five-level  simple  masks 
compensate  for  taa  lower  visual  acuity  in  the  diagonal  directions 
by  weighting  tae  diagonal  elements  highar  in  the  corresponding 
masks. 

(f)  Tha  structure  and  integer  weights  of  tha  simple  masks  make 
them  especially  suitable  for  fast  . computation  of  gradient 
magnitudes  and  diractions  digitally. 


Edge  Detection  System:  The  block  diagram  of  the  proposed  edge 
detection  system  is  shown  in  figure  3.  The  application  of  the  first 
four  simple  masks  to  a 3 x 3 grid  surrounding  a picture  element  gives 
the  gradient  magnitada  and  direction.  The  gradient  picture  is 
obtained  by  taking  taa  maximum  gradient  value  at  each  point.  The  mask 
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which  yields  the  eaxieua  gradient  value  determines  the  direction  of 
the  edge.  The  edge  map  is  a two  diaensional  array  of  nuabers  which 
range  between  0 and  7.  The  threshold  aap  can  be  obtained  in  various 
ways.  In  general  it  is  a two  diaensional  array  which  is  used  in 
deteraining  whether  the  gradient  value  is  large  enough  to  accept  or 
reject  the  presence  of  an  edge  point.  The  presence  of  an  edge  is 
deterained  by  ezaainiag  the  gradient  values,  edge  direction  aap  and 
the  threshold  aap.  It  the  edges  in  a 3 x 3 grid  surrounding  a point 
satisfy  the  local  connectivity  conditions  depicted  in  figure  4,  and  if 
they  are  above  the  threshold  set  by  the  threshold  aap#  then  it  is 
deterained  that  there  is  an  edge  point,  k binary  edge  aap  is  thus 
generated  at  the  output. 

Figure  5 shows  3oaa  exaaples  of  analog  gradient  inages.  Figure  6 
shows  the  iaportance  of  the  three  basic  blocks  of  the  edge  detection 
system  for  extracting  tae  edges  in  a picture  of  a toy  tank.  The 

] 

comparison  of  the  results  with  the  results  of  the  Rirsch  and  Sobel 
operators  shows  that  the  use  of  the  local  connectivity  and  the  locally 
adaptive  threshold  is  necessary  to  inprove  the  extraction  of  visually 
significiant  edges.  Figure  7 shows  the  binary  edge  naps  of  two  other 
iaages  using  a fixed  threshold  egual  to  the  average  intensity  of  the 
corresponding  analog  gradient  inages. 

Edge  activity  Ialac  and  Locally  Adaptive  Threshold:  Exanination 
of  the  analog  gradient  iaages  shown  in  figures  5,  6 and  7 shows  that, 
a suitable  fixed  thraahold  value  would  produce  the  edges  directly. 

However,  it  should  be  observed  that  soae  edges  are  very  faint,  while 


Original  Images 


Analog  Gradient  Linages 


Figure  2.3-5.  Some  examples  of  Analog  Gradient  Images. 
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soae  are  stronger.  If  the  threshold  is  too  low,  too  may  edge  points 
are  obtained  and  if  the  threshold  is  too  high,  then  soae  significant 
edges  aca  lost.  This  suggest  that  iaproveaent  of  th*  analog  gradient 
iaage  and  the  use  of  a Local  threshold  is  necessary. 

The  iaproveaent  of  the  analog  gradient  iaage  caa  be  obtained  by 
the  use  of  the  elge  activity  index  BAI  defined  as  the  ratio  of  the 
aaxiaun  gradient  nagnitude  at  an  iaage  point  to  the  average  aagnitude 
of  gradients  in  eight  coapas3  directions.  If  the  eight  directional 
gradient  values  at  a pixel  (i,  j)  are  y -j  ,then 


VI ax  f I V I.  lr  = 0.  1 7} 


could  serve  as  an  obj active  seasare  of  busyness  on  a 3 z 3 grid  of  an 
iaage.  If  the  EAI  is  greater  than  a threshold,  i.e.  if  the  edge 
activity  is  considerably  superior  in  the  direction  of  the  naxiaun 
gradient,  then  the  aaxiaua  gradient  value  is  taken,  otherwise  the 
gradient  value  is  set  to  zero.  This  operation  results  in  a sharper 
histograa  for  the  analog  gradient  ieage.  It  has  been  experiaentally 
verified  that  the  gradient  iaage  has  a Hayleigh-like  distribution. 
The  histograa  is  vary  peaked  for  a large  class  of  natural  grey  tone 
iaages.  The  shape  of  the  histograa  does  not  change  for  various  types 
of  directional  easts,  k siaple  value  for  a fixed  threshold  value  in 
edge  detection  is  the  aaan  intensity  of  the  gradient  iaage.  In  the 
exaaples  edge  aaps  generated  with  fixed  threshold  are  obtained  using 
the  aean  intensity  as  tae  threshold  on  the  corresponding  gradient 
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inage,  in  addition  to  tae  local  connectivity  tests  on  the 
corresponding  edge  direction  nap.  Another  choice  of  the  fixed 
threshold  can  be  obtained  by  integrating  the  histograa  starting  froa 
the  high  intensities*  since  the  edges  have  high  intensity  values  in 
the  gradient  isage.  Sore  and  sore  edge  points  can  be  added  in  order 
to  attain  a fixed  nuaoec  of  edge  points  in  the  resultant  binary  edge 
nap. 


A locally  adaptive  threshold  has  been  obtained  by  coaparing  the 
analog  gradient  isage  with  a blurred  version  of  the  original  isage, 
which  is  obtained  by  a low-pass  operation  on  the  isage.  The 
particular  low  pass  operation  can  be  perforael  by  the  sash  HQ.  Thus 
the  locally  adaptive  threshold  has  been  defined  as 

MaxCly^l,  k * 0,1,..., 7) 

LAT  = 

Output  of  tile  lovrpass  filter  MQ  at  pixel  (i,  j) 

Figure  6 shows  the  result  of  the  use  of  a locally  adaptive  threshold 
on  the  toy  tank. 

Coding : Prelisinary  results  show  that  the  transnission  of  edges 
extracted  using  coapass  gradient  nasks  say  be  enough  for  sone  low 
level  applications,  for  aore  sophisticated  applications,  a staple 
grey  tone  isage  caa  be  generated  by  transnitting  a reduced  size 
version  of  the  original  inage.  The  size  reduction  is  obtained  by 
successive  size  halving  of  the  input  inage  by  averaging  four 
neighboring  picture  dissents  on  two  lines.  The  reduced  size  inage 


Page  56 


(32  x 32  or  64  x 64  for  exasple)  can  os  coded  by  standard 
two-diaensional  coding  techniques.  At  the  receiver  a full  size 
blurred  ieage  is  foraed  by  successive  linear  interpolation.  The 
superposition  of  tha  edges  with  this  grey  tone  ieage  is  expected  to 
give  eore  visual  inforiation  than  just  an  edge  inage,  while  providing 
a good  data  rate  reduction.  Investigation  of  efficient  coding  eethods 
as  well  as  receiver  interpolation  of  edge  intensities  is  presently 
under  way. 
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2.4  Color  Edge  Detection 
Guner  S.  Robinson 

Color  isages  nay  be  described  quantitatively  at  each  pixel  by  a 
set  of  three  tristiiulus  values  which  ace  proportional  to  the  asount 
of  red,  green,  and  blue  priaary  lights  required  to  natch  the  pixel 


color.  The  red,  graan  and  blue  coaponenta  any  be  transforaed  by  soae 
linear  or  nonlinear  invartible  function  to  quantities  which  are  aore 
suitable  for  efficient  transaission,  color  perception,  redundancy 
reduction,  edge  extraction,  etc.  <1>.  The  I.T. S.C.  transaission 
color  coordinates  1,  l and  Q are  obtained  froa  the  tristiaulus  color 
coaponents  by  a linear  tcansforaation.  The  T coaponent,  which  is  a 
weighted  sua  of  the  tristianlus  values,  is  a aaasura  of  the  luainance 
of  the  color.  The  I and  Q coaponents  jointly  describe  the  hue  and 
saturation  of  the  iaage. 
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Several  definitions  of  color  edges  are  possible,  depending  on  the 
choice  of  the  color  coordinate  systea.  In  ■onochroae  iaages  an  edge 
is  defined  as  a discontinuity  in  the  brightness  function.  A 
generalization  of  tiis  definition  would  be  possible  by  defining  a 
discontinuity  in  the  bhcee-diaensiona 1 vector  space  defined  by  R,  G,  B 
or  other  color  coordinates  obtained  froa  S , G,  B by  a linear  or 
nonlinear  transf oraation . For  exaaple  an  edge  in  a color  iaage  could 
be  defined  as  a discontinuity  in  the  luainance  coaponent,  ignoring  the 
discontinuities  in  I and  Q in  regions  of  constant  luainance.  Then  the 
color  edga  detection  problea  reduces  to  the  case  of  aonochroae  edge 
detection.  It  is  known  that  the  red,  green  and  blue  coaponents  of  an 
iaage  are  highly  correlated.  The  transf oraation  which  gives  the 
N.T.S.C.  color  transaission  coordinates  partially  reaoves  this 
correlation.,  i.e.  the  Y,  I,  Q coaponents  ara  still  partially 
correlated.  A second  approach  to  tha  color  edge  detection  problea 
involves  use  of  the  Kit hunen-Loeve  transf oraation  to  decorrelate  the 
color  coaponents.  Edge  detection  is  than  perforaed  on  the  individual 
principal  coaponents  oC  the  particular  iaage.  Still  another  approach 
is  to  perfora  the  edge  detection  operation  on  each  coaponent  of  the 
color  space  and  coabine  the  edges  suitably,  depending  on  the 
definition  of  the  color  coordinates.  This  approach  then  reduces  the 
problea  of  color  edge  detection  to  repetition  of  the  aonochroae  edge 
detection  schene  for  all  three  coaponents.  The  0 V V and  Lab  color 
coordinate  systeas  could  also  be  used  for  color  edge  detection  since 
they  posses  a aetcic  color  difference  sensitivity  to  a good 
approxiaation,  while  tha  R,  G,  B space  does  not  <1>. 
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In  order  to  substantiate  the  above  ideas  and  conjectures,  various 
color  coaponents  have  been  coaputed  using  the  rad,  green  and  blue 
coaponents  of  a 256  x 256  color  iaage  shown  in  figura  1.  The  sets  of 
color  coaponents  that  were  considered  are:  principal  coaponents 
PjP2P3,  I I Q,  La  b,  and  Gj  G y The  coaponents  Gj,  G2  and  °3  are 
obtained  using  the  aodel  of  the  huaan  visual  systea  developed  by  Frei 
<2>.  Figures  2 to  5 snow  the  various  coaponents,  which  are  scaled  to 
have  the  saae  rang  a of  values,  0-255,  for  display.  The  percentage 
energy  each  coaponent  represents  is  given  in  Table  1.  The  nornnlized 
covariance  aatrix  of  the  red,  green  and  blue  coaponents  of  the  girl 
picture  is 


The  principal  coaponaats  have  been  obtained  using  the  above  covariance 
aatrix. 

Bxperiaental  Basalts  for  Color  Edge  Detection:  Experimental 
results  in  the  edge  detection  are  confined  to  tka  directional  sash 
edge  extraction  algoritaa  <3>.<  The  directional  sash  edge  extraction 
algoriths  was  applied  to  each  coaponent  separately.  The  binary  edge 
saps  vers  generated  by  using  local  connectivity  and  a fixed  threshold. 
The  threshold  in  each  case  was  the  seen  intensity  of  the  analog 
gradient  picture.  The  color  edge  was  obtained  sisply  by  conparing  the 


Color  Component 

Range  of  Values 
(min- max) 

Energy  Content 
(%) 

R 

56.0 

255.0 

37.6 

G 

3.0 

245.0 

43.8 

B 

43.0 

219.0 

18.6 

P1 

59.2 

400.  0 

90.6 

P2 

-91.4 

60.4 

8.4 

P3 

24.7 

101.2 

0.8 

Y 

25.9 

241.6 

89.2 

I 

14.9 

88.0 

10.3 

Q 

25.8 

66.  0 

0.  5 

L 

38.2 

98.0 

82.  1 

a 

- 1.4 

61.0 

16.8 

b 

-30.9 

34.3 

1.  1 

G1 

70.0 

118.0 

96.9 

G2 

-17.5 

1.76 

2.9 

G3 

- 2.89 

6.05 

0.  2 

Table  1.  The  range  of  values  and  the  percentage  energy- 
content  of  various  color  components. 


N.  T.  S.  C.  Color  Transmission 
Components  of  a Color  Image. 
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twenty  four  gradient  values  in  three  coaponents  and  eight  directions. 
The  gradient  that  gives  the  aaxiaua  value  at  a pixel  thus  detersines 
the  direction  of  the  adge,  whether  it  is  due  to  a discontinuity  in  the 
first,  second  or  the  third  color  coapoaent.  Figure  6 shows  soae 
exaaples  of  gradient  iaages  of  color  coaponents  and  figures  7 and  8 
illustrate  exaaples  of  the  color  edge  extraction  algoritha. 

Comparison  of  Color  Coordinates:  Visual  exaaination  of  the  color 


coaponents  shown  in  figures  2 to  5 shows  that  the  conjectures  that 
were  Bade  in  the  selection  of  five  sets  of  color  coordinates  are 
basically  correct.  Tae  percentage  energy  each  coaponent  represents 
is,  in  general,  indicative  of  the  edge  activity  in  that  coaponent. 
The  principal  coaponents  do  not  have  a physical  Banning  in  terns  of 
the  perception  of  colors.  However,  the  first  principal  coaponent  is  a 
weighted  sun  of  the  tristiaulus  values,  and  therefora,  is  a aeasure  of 
the  brightness  of  the  image  similar  to  the  T anl  L components.  The 
first  principal  coaponent  has  the  aost  energy  content,  which  is  aost 
likely  due  to  the  sharper  amplitude  discontinuities.  In  the  case  of 
the  Lab  components,  "L"  reflects  the  brightness  of  the  image,  the 
values  of  "a"  anl  "b"  reflect  the  redness-greenness  and 
yellowness-blueness  of  the  image,  respectively. 


Objective  measuras  in  addition  to  the  subjective  evaluation  are 
necessary  for  coaparing  color  components  for  suitability  for  edge 
extraction.  The  edge  activity  index  EAZ  could  be  used  as  an  objective 
measure  of  edge  busyness  in  an  image  <3>.  If  the  edge  direction 
angles  at  a point  (i,  j)  are  quantized  to  eight  coapass  directions,  the 
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BAI  provides  a sea  sore  of  tbs  superiority  of  the  sexless  gradient 
direction  to  the  gradients  in  the  other  seven  cospass  directions.  If 
the  values  of  BAI  are  cospotsd  os  the  edgs  positions  throughout  the 
isage,  then  the  sean  and  the  variance  of  the  BAI  is  expected  to  be 
indicative  of  the  elge  activity  in  that  isage.  The  study  of  the  edge 
activity  index  as  an  objective  aeasure  in  coopering  the  above  five 
sets  of  color  conpoaents  is  presently  under  vay. 

References 

1.  w.jc.  Pratt,  Digital  Isage  Processing,  Wiley,  Wes  fork,  1977. 

2.  W.  Frai,  "A  Quantitative  Model  of  Color  Vision,"  USCIP1  Report 
540,  Oniversity  of  Southern  California,  pp.  69-83. 

3.  G.S.  Robinson,  "Detection  and  Coding  of  Edges  using  Directional 
Masks,"  in  this  report. 

2.5  Hueckel  Color  Edge  detector 
Ran  Nevatia 

A coaeon  operation  perforaed  on  isages  is  that  of  edge  detection. 
Edges  are  defined  as  a discontinuity  in  soae  isage  attribute,  usually 
luninanca.  Boundaries  Between  two  objects  are  often  accoapanied  by  a 
discontinuity  in  luaiaance.  In  color  iaages  different  objects  nay 
have  different  colors  and  it  seeas  useful  to  look  for  color 
discontinuities  or  color  edges.  This  paper  provides  a definition  of  a 
color  edge  as  a generalization  of  the  Hueckel  edge  operator  <1-2>,  and 
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a technique  for  ita  computation.  Sona  axpariaaatal  results  ace 

pcwtDtad. 

Representation  of  a Color  Iaaga:  i digital  iaaga  is  dsscribod  by 
a function,  say  I(ity| , defiaad  at  cbossa  grid  points  of  tbs  inags. 
For  a gray  level  (acbcoaatic)  iaaga,  tbs  fssctioa  I is  scalar  valued; 
its  value  being  tba  luaiaaaca  of  tbs  iaaga  at  a certain  point.  For 
color  iaages,  three  values  oust  be  specified  at  each  point,  i.e.  tbs 
function  1 is  vector  valued  and  ban  three  coaponents. 

k coaaon  choice  for  the  three  coaponents  of  I is  the  Bed,  Green 
and  Blue  (R, G, B)  coaponents  of  a scene  vieved  through  red,  green  and 
blue  filters.  The  R,  5 and  B coaponents  can  be  transforaed  to  other 
quantities,  sore  closely  associated  with  buaan  visual  sensations  of 
color,  such  as  brightness,  hue  and  saturation.  The  choice  of 
coaponents  for  representing  color  is  discussed  later. 

Definition  of  a -oLor  Edge:  in  edge  in  an  achroaatic  inage  is 
defined  by  a discontinuity  in  the  luainanca  function,  1.  The  precise 
definition  of  a discontinuity  varies  with  the  different  edge 
operators.  To  detecaine  an  edge  in  a color  inage,  a definition  for 
discontinuity  in  a thcao-diaensional  color  space  is  required.  The 
siaplest  scheae  is  porhaps  to  conpute  edges  in  the  three  color 
coaponents  separately  and  deteraine  an  edge  in  the  color  inage  if 
certain  relations  between  edges  in  individual  coaponents  are 
satisfied.  This  paper  describes  a technique  in  which  a constraint  is 
inposed  before  the  coaputation  of  individual  edges.  This  approach  is 
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based  on  an  edge  operatic  developed  by  Hueckel  <2>  foe  a single,  grey 
level  iaage  which  is  ficst  described  briefly. 

The  tiuecknl  operator  determines  the  presence  of  an  edge  in  a 
circular  neighborhood  by  fitting  an  optiaal  step  to  the  input  signal. 

An  ideal  edge  step  is  defined  to  have  a step  or  line  profile  in 
aaplitude,  this  profile  being  constant  along  a straight  line  in  the 
signal  window.  The  edge  is  characterized  by  the  position  and 

orientation  a of  this  line,  and  six  other  paraaeters  which  deteraine 
the  aaplitude  and  the  profile  of  the  edge  (for  details,  see  <2>) . 
dueckel's  method  determines  the  paraaetars  of  the  edge  step  that 
Batches  best  with  the  iaage  in  a given  neighborhood. 

The  aatching  is  simplified  by  expanding  both  the  input  signal  and 
an  ideal  step  into  the  first  eight  eleaents  of  a series  of  orthogonal 
functions.  Let  the  coefficients  of  expansion  be  ai  and  si  for 
0 < i < 7.  Vote  that  the  s.  are  paraaeterized  by  the  edge  paraaeters. 
Optiaal  edge  paraaeters  are  found  by  ainiaizing 

7 

N2=^(a.-8.)2  (2.5-1) 

1=0 

It  is  shown  in  <2>  that  the  optiaal  value  for  the  edga  direction  a can 
be  detecained  independent  of  the  other  edge  paraaeters  as  a function 
of  coefficients  a.  . The  remaining  edge  paraaeters  are  then  coaputed 
separately.  An  edge  is  said  to  be  present  if  the  aaplitude  of  the 
atep  with  the  best  fit  is  sufficiently  large  and  the  fit  is 
sufficiently  good  (■  is  saall  coapared  to  the  step  aaplitude). 
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To  extend  this  concept  of  an  edge  to  a color  edge,  an  optical 
step  in  the  vector  color  space  is  defined  first.  Let  a step  in  the 
color  space  have  thraa  components  SR,  S G and  S B in  the  three  color 
components,  and  let  tae  corresponding  degrees  of  fit  he  H^,  Mq  and 
w her  e 


(2.5-2) 


Similar  equations  exist  for  fl_and  low,  an  optimal  step  in  color 

Cj  r> 

space  is  defined  by  determining  SR,  and  Sfi  such  that 


N2  = N_2  + N 2 + N 2 


(2.  5-3) 


is  minimized.  If  tae  three  step  components  are  allowed  to  be 
independent,  then  minimizing  R in  eg. (3)  is  equivalent  to  minimizing 
the  three  components  separately. 


To  simplify  the  optimization  the  three  components  are  allowed  to 
be  independent,  except  that  the  spatial  angles  are  constrained  to  be 
identical  (i.e.  a„  * a_  - a_ ) . Once  an  optimal  step  with  this 
constraint  is  obtained,  a decision  process  for  judging  the  presence  of 
an  edge  is  needed.  The  decision  may  be  based  on  the  degree  of  fit, 
determined  by  V,  and  a weighted  sum  of  the  squares  of  amplitudes  of 
the  steps  in  the  thraa  components,  analagous  to  the  achromatic  image 
case. 


Alternatively,  tae  presence  of  edges  in  the  three  components  nay 
be  determined  separately  from  components  of  ths  optimal  step  above. 
Presence  of  an  edga  in  the  color  image  may  eon  be  based  on 
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relationships  between  tae  edges  ia  the  three  coaponents.  Experimental 
results  using  two  types  of  relations  are  presented  later. 


Coaputation  of  a Calor  Edge;  An  expression  for  N,  defined  in 
eg.  (3)  can  be  easily  generated  from  expressions  for  individual 
coaponents  (following  eg.  (12)  in  <2>) . Froa  this  expression,  it  turns 
out  that  the  angle  a can  still  be  chosen  independent  of  other  edge 
paraaeters  (see  Appendix  1).  The  reaaining  edge  paraaeters  aay  now  be 
coaputed  exactly  as  for  the  black  and  white  iaages. 


Coaputing  the  optiaal  value  of  a requires  the  solution  of  an 
eighth  order  polynoaial.  As  an  approximation,  a can  be  determined  as 
a weighted  average  (weighted  by  the  step  amplitudes)  of  the  optimal 
angles  of  steps  in  the  three  components,  deterained  separately.  This 
is  the  method  used  for  the  results  presented  later. 


In  summary,  an  optimal  step  in  color  space  is  computed  as 
follows.  First,  optiial  steps  are  coaputed  for  the  three  coaponents 
separately.  A common  angle  is  deterained  by  a weighted  average  of  the 
three  angles.  Osing  this  new  angle,  the  other  edge  paraaeters  are 
recomputed,  each  component  being  treated  independently  again.  Also 
new  values  for  N^,  «G  and  HQ  are  obtained.  The  resulting  edge 
paraaeters  are  taken  to  define  an  optiaal  step  in  the  color  space. 
Next,  the  presence  of  edges  in  the  three  components  of  this  optimal 
step  is  detected  separately.  Relationships  between  these  edges  are 
used  to  infer  the  presence  of  color  edges  as  described  below. 


I 


Page  75 


|PL  ■ ,u~  ,L 


i 


Experimental  Results:  Figures  la#  1b  and  1:  show  the  R,  G and  B 
components  of  a coLor  picture  of  a girl,  an  SHPTB  (Society  of  Notion 
Pictures  and  Television  Engineers)  test  picture.  Pigures  2a  and  2b 
show  the  color  edgas  obtained  by  computing  an  optimal  step  in  color 
space  as  described  earlier.  Figure  2a  shows  tha  edges  that  occur  in 
at  least  two  conpoaaats  of  the  color  step  (the  displayed  edges  are 
those  occuring  in  the  strongest  component) . Figure  2b  shows  edges 
occurring  in  all  threa  components.  These  results  may  be  coapared  with 
the  edges  obtained  from  an  achromatic  image  derived  from  the  color 
girl  image  and  shown  in  figure  2c. 

The  edges  shown  in  figure  2a  contain  fewer  undesired  edges. 
However,  some  desired  edges  nay  be  lost  because  of  the  strict 
requirement  of  the  presence  of  edges  in  all  three  components.  Figure 
2b  contains  a larger  number  of  edges,  filling  in  some  of  tile  gaps  in 
figure  2a. 

The  number  of  components  occurring  in  a color  edge  nay  be 
considered  as  a measure  of  the  confidence  or  prominence  of  the  edge. 
In  subsequant  processing  only  edges  present  in  all  three  components 
nay  be  processed  initially.  Edges  with  fever  components  may  then  be 
used  to  fill  in  as  analysis  proceeds  and  more  context  becomes 
available. 

Another  approach  is  to  accept  a color  edge  if  an  edge  is  present 
in  any  of  its  color  components.  The  color  edge  is  further 
characterized  by  the  components  in  which  edges  are  present.  This 
description  is  used  ia  further  processing  of  tha  edge  data. 

hi  ii«r  
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(b)"G"  component 


(c)  "B"  component 


(a)  "R"  component 


Figure  2.  5-1.  Three  components  of  a color  picture 


(c)  Edges  in  an  achromatic  (d)  Color  edges  linked 

image  into  segments 


Figure  2.  5-2.  Edges  detected  in  pictures  of  figure  1. 
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A coaaon  step  ia  the  processing  of  an  inaga  is  to  link  the  local 
edges  into  longer  segaents.  One  technique  of  linking  edges  into 
nearly  straight  line  sagaents  ia  described  in  <3>.  It  is  also 
suggested  there  that  the  elongated  segaents  are  often  useful  in 
isolating  the  edges  b alaaging  to  object  boundaries.  Only  neighboring 
edges  having  orientations  sithin  specified  liaits  of  each  other  are 
linked.  Further#  only  edge  segaents  that  contain  at  least  a certain 
aininua  auaber  of  edge  eleaents  are  preserved. 

The  description  of  a color  edge  as  defined  above  allows  further 
constraints  to  be  iaposed  on  linking  of  two  edges.  The  two  edges  to 
be  linked  are  required  to  have  coapatible  color  characteristics.  Two 
edges  are  defined  to  be  color  coapatible#  if  the  color  coaponents  in 
which  one  edge  occurs  are  the  saae  as#  or  a subset  of#  the  coaponents 
in  which  the  other  edge  occurs.  Figure  2d  shows  the  result  of  linking 
edges  obtained  froa  tbe  girl  picture  with  this  additional  constraint 
and  using  the  aetaod  described  in  <3>.  Only  segaents  consisting  of 
aore  than  five  edge  eleaents  are  shown.  Again#  linking  with  tight 
constraints  nay  be  used  to  obtain  boundary  segaents  initially.  These 
constraints  nay  then  be  relaxed  based  on  the  additional  context 
available  now. 

Choice  of  Color  Coordinates;  The  edge  detection  scheae  described 
above  i3  applicable  to  any  inage  where  each  pixel  is  described  by 
three  nuabers.  The  conventional  R#  S and  B coaponents  of  an  iaage 
were  used  for  the  results  presented  in  the  previous  section.  These 
coaponents  do  not  correspond  directly  to  the  hunan  perceived  color 
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attributes  of  an  iaage.  Hunan  perception  of  color  is  normally 
describe!  by  the  attributes  of  brightness,  hue  and  saturation.  The  R, 
G and  B components  can  be  transforned  to  give  these  attributes.  Bdges 
obtained  in  the  transfocaed  space  have  sore  intuitive  aeaning  and  say 
be  easier  to  use  in  further  processing,  e.g.  it  nay  be  useful  to 
extract  edges  differing  in  hue  only. 

f 

Edge  detection  experiments,  similar  to  those  described  for  the 
R,G  and  B components,  were  performed  in  a transforned  color  space 
using  a transformation  based  on  a model  of  human  visual  system 
developed  by  Frei  O > . The  transforned  space  contains  a luminance 
component  and  two  chromaticity  components.  The  chroaaticity 
components  are  claiaai  to  be  independent  of  the  luminance  for  a vide 
range.  The  three  components  were  scaled  to  have  the  sane  range  of 
values.  The  results  obtained  are  comparable  to  those  of  figures  2a, 
2b  and  2d.  However,  hare  the  chroaaticity  edges  are  separated  from 
the  luminance  edges  and  the  liuking  of  edges  can  be  more  selective. 

Conclusions:  An  optimal  step  in  color  3pace  that  allows  the  steps 
in  the  three  components  to  be  combined  in  useful  ways  has  been 
defined.  Some  alternatives  for  combining  the  components  were 
discussed  and  experiaantal  results  presented.  An  important  issue 
requiring  further  experimental  investigation  is  the  choice  of  color 
space  used  to  represent  an  image. 


Acknowledgements:  The  definition  of  optimal  color  step  presented 
here  was  suggested  to  the  author  by  Or.  T.O.  Binford.  U.D.  Hiller 
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helped  with  the  early  analytical  and  prograaalog  work.  Dr.  H.  prei 
provided  each  useful  iaforaatioa  on  the  choice  of  color  coordinates. 

Appendix  1 

In  reference  <2>,  Hueckal  presents  a formula,  eg.  (12) , for 
computing  I (for  aa  achromatic  image)  which  may  bo  abbreviated  as 


follows 


N'fj<*>  ♦ f2(«)+  fj(T)  (2 

where  fj  is  a function  of  the  coefficients  a.  , f , is  a function  of  a 
only  and  f3  is  a function  of  vector  jl  containing  all  the  edge 
paraaeters.  It  is  saowa  in  <2>  that  f 3 is  always  agual  to  zero  at  the 
optiaal  solution  point,  and  that  to  sininize  I,  a nay  be  chosen 
independent  of  other  parameters  in 

For  the  color  case  it  is  necessary  to  niniaize 

N*  * 2 V*> + vo)  )Z  <* 

J=R,G,B 

Differentiation  with  respect  to  a yields  an  eighth  order  polynomial  in 
cos(a).  After  a is  coaputed,  other  paraaeters  of  _yR,  2^  and  jt^can  be 
coaputed  separately  a3  for  the  case  of  black  ani  white  inages. 
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2.6  Becursive  Bstinator  for  the  Deteraination  of  Boundaries 
Basser  E.  Bahi  and  Sison  Lopez-Mora 

The  purpose  of  tais  research  is  to  develop  recursive  algorithss 
for  the  estieation  of  boundaries  of  objects  Ln  noisy  pictures.  The 
available  inforsation  is  asaueed  to  be  liaited  to  the  statistics  of 
the  object  boundary  foreground  and  background. 

The  scanned  picture  is  represented  by 

i 

e(k)  = X(k)  «o(k)  + (1-X(k))eb(k)  (2.6-1) 

2 

for  k « 1, . • . ,N  where  si  is  the  nusber  of  lines  in  the  picture,  sq  and 
sb  denote  the  intensity  values  of  object  and  background,  assused  to  be 
sanple  functions  of  tea  statistically  indepenlent,  cyclo-stationary 
randoe  sequences  whose  first  two  soeents  are  available,  and  X is  a 
binary  valued  function  taking  values  of  1 or  0 corresponding  to  points 
of  the  inage  belonginj  to  the  object  or  the  background  respectively. 

The  doaains  of  sQ,  sb  and  X correspond  to  the  entire  picture. 
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K set  of  observations 

r(k)  = s(k)  + v(k)  (2.6-2) 

is  assusad,  with  s (k|  as  defined  in  »q.  (1)  and  v(k)  representing  a 

2 

zero  neau  Gaussian  white  noise  sequence  of  variince  o . 


The  dyuanic  systen  chosen  to  represent  X(k)  is  given  by 


(i)  ♦ B(i)  uj  (i) 
u2 


(2. 6- 3s) 


Mk)  = .5(1+  sign  (w  (l)  + w)  - c - TRl(k  - 'aft)-  •))} 

l*(l-l)N+UksiNSN2 


where. 


I 

sign  (s)  s 0 
-I 


if  s > 0 
if  s « 0 
if  s < 0 


TRl(s)  = 


if  .&  s .<  0 

s 

“ * * »«f 


TRl(n  + N)  * TR1  <s)  for  aU  s 


(2.6.3b) 


(2.6-4e) 


(2.6.4b) 


(2. 6.4c) 


for  c > 0.  It  is  shown  that  this  nonlinear  nodel  represents  the 

statistics  of  x , waern  w(i|  is  tho  variation  of  ths  object  width  at 

line  i about  its  aeaa  T,  s(i)  is  the  deviation  of  the  width  center 

froa  its  aean  s (aeasuced  froa  the  aiddle  of  the  picture),  A(i),  B(i) 

T 

are  aatrices  of  appropriate  dinansions,  and  u.  = (u1#u2)  is  a zero 
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oean  white  Gaussian  sequence  of  unit  covariance. 


When  dynaaic  Bodal3  representing  the  statistics  of  the  object  and 
the  background  are  adjoined  to  eg.  (3)  it  is  possible  to  foranlate  a 
nonlinear  ainiaua  awaa  aguare  error  estiaation  problea  for  which  the 
aethod  introduced  by  Nahi  and  Maraghi  <1>  can  be  iapleaented.  To 
illustrate  the  perforaaace  of  the  estiaator,  a binary  picture  with  an 
object  intensity  of  unity  and  a background  intensity  of  xero  has  been 
used.  The  pnraaeters  for  the  dinaond  shape  in  eg. (3)  were  calculated 
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with  Y ■ iS,  Y • 0#  f * 256  and  c ■ 1.  To  siayle  oat  the  boundary 
estiaation  itself , s0(c|  ■ 1 and  sb(k)  * 0 hate  been  ased.  Mote  that 
Y « 0 correspoads  to  objects  that  should  be  centered,  etea  though  the 
original  picture  ia  not.  Figure  la  ahowa  tha  observation  seguence 
when  Gaussian  noise  of  standard  deviation  0.5  has  beaa  added  to  the 
original  picture  whose  object  intensity  is  unity  sad  with  background 
intensity  xero  while  figure  1c  shows  the  estiaated  (brighter  trace) 
and  original  boundaries.  Siailar  results  far  the  case  of  Gaussian 
noise  with  variance  a appear  in  figures  1b  and  Id. 


Further  work  is  onlerway  to  generalise  these  results  to  sore 


(c)  Estimated  and  original 
boundaries 


(d)  Estimated  and  original 
boundaries 


Figure  2.  6- 1 Recursive  estimator  for  bounda 


ries. 
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conplex  shaped  objects  possessing  multiple  gray  levels. 
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2.7  Figure  of  Merit  for  Edge  Location 
Hilliaa  K.  Pratt 


Relatively  few  studies  of  edge  detector  perforaance  have  been 
reported  in  the  literature  <1,2>.  A perforaance  evaluation  is 
difficult  because  of  the  large  nuaber  of  proposed  aethods, 
difficulties  in  detaraining  the  best  paranetars  associated  with  each 
technique,  and  the  lack  of  definitive  perfornance  criteria. 

In  developing  pacforaanca  criteria  for  an  edge  detector,  it  is 
vise  to  distinguish  between  nandatory  and  auxiliary  infornation  to  be 
obtained  froa  the  detector.  Obviously,  it  is  absolutely  essential  to 
deteraine  the  pixel  location  of  an  edge.  Other  inforaation  of 
interest  includes  the  height  and  slope  angle  of  the  edge  as  well  as 
its  spatial  orientation.  Another  useful  iten  is  a confidence  factor 
associated  with  the  edge  decision,  for  exaaple,  the  closeness  of  fit 
between  actual  inage  data  and  an  idealised  edge  nodal.  Oafortuaately, 
few  edge  detectors  provide  this  full  gaaut  of  inforaation. 


I 


« 


There  are  three  eajor  types  of  error  associated  with 
determination  of  aa  edge  location:  (a),  missing  valid  edge  points; 
(b) , non- localized  eige  points;  (c) , improper  classification  of  noise 
pulses  as  edge  points.  Figure  1 illustrates  a typical  edge  segaent  in 
a digital  image,  in  ideal  edge  representation,  and  edge 


representations  subject  to  various  types  of  error. 


A coaaon  strategy  in  signal  detection  problems  is  to  establish 
sose  bound  on  the  probability  of  false  detection  resulting  froa  noise, 
and  then  atteapt  to  maximize  the  probability  of  true  signal  detection. 
Extending  this  cone  apt  to  edge  detection  simply  involves  the  setting 
of  the  edge  detection  threshold  at  a level  3uch  that  the  probability 
of  false  detection  rasulting  froa  noise  alone  does  not  exceed  some 
desired  value.  The  probability  of  true  edge  detection  can  be  readily 
evaluated  by  a coincidence  conparison  of  the  edge  naps  of  an  ideal  and 
an  actual  edge  detector.  The  penalty  for  non-localized  edges  is 
somewhat  more  difficult  to  assess.  Edge  detectors  which  provide  a 
smeared  edge  location  saould  clearly  be  penalized,  however,  credit 
should  be  given  to  edge  detectors  whose  edge  locations  are  localized 
but  biased  by  a snail  aaount. 


An  edge  location  figure  of  aerit  for  digital  images  has  been 
developed.  The  figure  of  merit  is  defined  as 


1 

1+ad2 


(2.7-1) 


where  IN*  max and  Ij  and  IA  represent  the  number  of  ideal  and 


t. 


Figure  2.7-1.  Indications  of  edge  location. 
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actual  edge  eap  paints,  a is  a scaling  constant  and  d is  the 
separation  distance  of  an  actual  edge  point  noraal  to  a line  of  ideal 

edge  points.  The  eating  factor  is  noraalizai  so  that  K * 1.0  for  a 

perfectly  detected  edge.  The  scaling  factor  a nay  be  adjusted  to 
penalize  edges  which  are  localised  but  offset  froa  the  true  position. 
Noraalization  by  the  aaciaua  of  the  actual  and  ideal  nuaber  of  edge 
points  insures  a penalty  for  sneered  or  fragaented  edges,  is  an 
exaaple  of  perforaanca,  if  a = 1/9,  the  rating  of  a vertical  detected 
edge  offset  by  one  pixel  is  set  at  B = 0.90,  and  a two  pixel  offset 

gives  a rating  of  R 5 0.69.  Kith  a = 1/9,  a sa eared  edge  of  three 

pixels  width  centered  about  the  true  vertical  edge  yields  a rating  of 
R = 0.93  and  a five  pixal  wide  saeared  edge  gives  R * 0.84.  A higher 
rating  for  a saeared  edge  than  for  an  offset  edge  is  reasonable 
because  it  is  possible  to  thin  the  saeared  edge  by  post-processing. 

Soae  edge  detectors  provide  an  indication  of  edge  height,  slope 
angle,  and  orientation.  For  these  detectors,  it  is  useful  to 

deteraine  the  Bean  sgaace  deviation  in  these  guantities  froa  their 
true  values  averaged  over  all  true  edge  points  which  are  coincident 
with  detected  edge  points. 

The  perforaance  evaluation  aethodology  described  above  has  been 

applied  to  the  assassaent  of  edge  detection  technigues  using  a test 

iaage  consisting  of  a 64  x 64  pixel  array  over  a 0 to  255  aaplitude 

range  with  a vertically  oriented  edge  of  variable  contrast  and  slope 

placed  at  its  center.  Independent  Gaussian  noise  of  standard 

2 

deviation  a has  naaa  added  to  the  edge  iaage,  and  the  reseltaat 
n 
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piztuce  has  been  clipped  to  the  aaziaaa  display  Halts  (0  to  255) . 
The  signal-to-noise  ratio  is  defined  as 


SNR  = 


(2. 7-2) 


where  h is  the  edge  height. 


Since  the  pnrposa  of  the  testing  is  to  coapare  the  perforaance  of 
various  edge  detectioo  aethods,  for  fairness#  it  is  iaportant  that 
each  edge  detector  be  tuned  to  its  best  capabilities.  Consequently# 
each  edge  detector  is  pernitted  to  train  both  on  randoa  noise  fields 
without  edges  and  the  actual  test  inages  prior  to  evaluation.  For 
each  edge  detector  the  threshold  paraaeter  is  set  to  achieve  the 
aaziaua  figure  of  aarit  subject  to  the  aaziaua  allowable  false 
detection  rate. 


Figure  2a  contains  a plot  of  the  figure  of  aerit  as  a function  of 
signal-to~noise  ratio  for  several  edge  detectors.  The  figure  of  nerit 
is  plotted  as  a function  of  edge  width  in  figura  2b.  The  figures  of 
aerit  ganarally  follow  expected  trends:  low  for  low  contrast#  vide# 
noisy  edges;  and  higa  in  the  opposite  case.  Sone  of  the  edge 
detection  aethods  are  universally  superior  to  others  for  all  test 
iaages. 

As  a check  on  tha  subjective  validity  of  tae  edge  location  figure 
of  aerit#  figures  3 and  4 present  the  edge  location  naps  obtained  for 
several  tast  iaages  for  high  and  low  ranking  edge  detectors.  These 
figures  tend  to  corroborate  the  utility  of  the  figure  of  aerit.  A 


FIGURE  OF  MERIT 


SIGNAL -TO -NOISE  RATIO 


(a)  Figure  of  merit  vs.  SNR 
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(b)  Figure  of  merit  vs.  edge  width 

Figure  2.7-2.  Edge  location  figure  of  merit  as  a function  of 
signal-to-noise  ratio  and  edge  width. 


Figure  2.  7- 3.  Edge  location  perfon 
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high  figure  of  aerit  generally  corresponds  to  a veil  located  edge  upon 
visual  analysis  and  vica  versa. 
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2.8  Extraction  of  Proainent  Features  froa  Aerial  Photographs 
Lee  H.  Hartin  and  Baa  Mevatia 


The  identification  of  "proainent  features"  in  an  iaage  yields 
guideposts  for  furthar  processing.  Proainent  features  are  those  that 
correspond  to  areas  of  interest  for  scene  analysis,  and  are  likely  to 
exhibit  soae  global  properties.  For  exaaple,  the  features  nay 
correspond  to  partial  boundaries  and  be  used  to  direct  the  scene 
segaentation  process. 


Tha  attributes  of  n useful  feature  are,  in  general,  unknown.  The 
ideas  upon  which  to  base  feature  extraction  am  founded  on  hunan 
observations  of  seen* a nod,  as  is  presently  the  case,  the  hunan 
observer  is  the  final  arbiter  of  what  constitutes  prominent  features. 
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In  this  report  tjatures  useful  for  Bating  correspondences  between 
| two  iaages  of  the  sans  scene  for  inage  registration  or  stereo  Batching 

are  considered.  This  approach  is  an  alternativa  to  previously  used 

M 

techniques  of  bull  area  correlation  which  are  very  expensive 
computationally  <1>.  extraction  of  soae  proainent  features  will 

* V J 

significantly  reduce  the  nuaber  of  eleaents  to  be  searched  for 

; 

correspondences.  Soaa  criteria  for  detaraining  features  are: 

(a)  relative  invariance  under  slightly  dissiailar  iaaging 
conditions  and  saall  perspective  changes. 

(b)  reproducibility  of  the  feature. 


(c)  uniqueness  or  at  least  the  set  of  features  which  have  coaaon 
attributes  should  be  3aall,  to  be  usefal  in  coapleaent  iaage 
Batching. 

(d)  correspondence  with  areas  of  interest  in  the  iaage,  e.g. 
object  boundaries. 


Thi3  section  describes  a technigue  for  extracting  features 
derived  froa  edge  data.  The  features  are  called  edge  segaents.  Local 
edges  consist  of  lints  satisfying  certain  relationships  of  proxiaity 
and  orientation.  Long  edge  segaents  often  correspond  to  object 
boundaries  <2>.  Such  edge  segaents  satisfy  the  criteria  above,  to  a 
degree,  and  are  proainent  in  the  sense  that  a huaan  observer  is  likely 
to  note  then  in  describing  the  scene.  A brief  description  of  the 
prograa  operation  follows. 
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A Hueckel  <3>  ei }b  operator  processes  the  antire  iaage  returning 
edge  data  on  position*  orientation/  and  type.  Edges  are  typed 
depending  upon  whether  the  discontinuity  within  the  view  area  of  the 
operator  is  nost  lika:  a step  (S) ; a line  ( L)  ; or  unclassif iable  as 
either  (U) . The  distinctions  are  described  briefly  below  (for  aore 
detail  refer  to  <3>). 

The  view  area  of  tne  edge  operator  is  guasi-circular*  and  in  the 
idealized  circle  a step  (S)  changes  froa  soae  average  brightness  bj  to 
a second  value  b2«  only  two  average  brightnesses  are  considered  in  the 
disk;  a line  (L)  occurs  when  there  is  a change  froa  bj  to  b2  and  back 
to  bj  within  the  disk.  The  unclassif iable  type  results  froa  edges 
which  cannot  be  typed  as  being  an  S edge  or  an  L edge. 

The  edge  data  obtained  is  operated  upon  by  a prograa  described  in 
<2>.  Briefly*  edge  points  which  have  an  orientation  along  a given 
direction  (with  a given  angular  tolerance)  are  considered.  Edges 
which  satisfy  certain  proxiaity  relations  are  linked  into  edge 
segaents.  These  edge  sagaents  are  ordered  sets  of  edge  points  which 
denote  connectivity  between  adjacent  edge  elenents.  Only  segaents 
which  contain  aore  tnan  n links  (here*  n > 5)  are  retained.  This 
process  is  repeated  for  several  angles  over  the  entire  range  of 
directions  in  snail  steps  (e.g.  here*  every  10  degrees). 

In  the  present  prograa*  linking  is  perforaad  on  any  of  the  three 
typed  edge  sets  or  coabinations  thereof  to  create  edge  segaents. 
Figure  1 shows  the  digitized  iaage  which  is  256  z 256  and  figure  2 
illustrates  typical  results.  The  total  nuaber  of  step  edge  points  is 


Figure  2.8-1.  Digital  aerial  image 

(Resolution  256  X 2 56) 


(b)  Step  edge  segments 
(Step  edge  prominent 
features) 
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(c)  Line  edge  points 
(1213  edge  points) 


(d)  Line  edge  segments 
(Line  edge  prominent 
features) 


Figure  2.  8-2.  Effects  of  edge  linking  to  extract  prominent  features 
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shown  in  figure  2a.  The  linked  segaents  which  result  are  shown  in 
figure  2b.  Siailar  results  are  shown  for  line  edges  in  figures  2c  and 
2d.  No  obviously  proiiaant  edges  were  observed  by  linking  over  the 
unclassified  edge  points  by  theaselves,  however,  when  the  step  and 
unclassifiable  edges  ware  unioned  and  linked  soae  boundary 
discontinuities  were  closed. 


The  linked  edge  segaents  are  features  with  direction 
(orientation) , location,  size,  shape,  and  position.  Note  the  effects 
which  occur  in  going  froa  the  data  set  of  figure  2a,  the  edge  points, 
to  that  of  figure  2b,  the  edge  segaents  (siailarly  for  figures  2c  and 
2d).  The  edge  points  which  are  retained  in  edge  segaents  correspond 
to  the  delineation  of  soae  of  the  proainent  features,  e.g.  outlining 
soae  river  and  lakeshora  areas,  as  well  as  auch  of  the  highway  in  the 
left  part  of  the  picture  of  figure  2b.  The  linking  of  line  edges  in 
figure  2d  indicates  the  presence  of  line  type  features  (the  highway  is 
picked  up  as  both  a step  edge  and  a line  edge  depending  upon  the 
placeaent  of  the  edge  operator)  . 

Even  though  noise  obliterates  soae  segaents  that  a huaan 
observes,  the  proainent  features  are  still  easily  recognizable.  This 
processing  is  only  ezpected  to  give  features  that  suggest  possible 
object  boundaries.  also  iaportant  is  the  reduction  of  the  nuaber  of 
eleaents  being  considered.  Not  only  have  approziaately  35%  to  50%  of 
the  edges  been  rejected  for  failure  to  aeet  linking  criteria,  but 
those  retained  are  in  sets  of  edge  points.  The  search  space  using 
edge  segaents  contains  considerably  fewer  eleaents.  Note  that  a 
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majority  of  edges  which  occur  due  to  texture  are  eliminated.  The 
edges  ia  textured  regions  hive  nndos  orientations  and  do  not  fora 
sufficiently  long  connected  segaents. 

Thus  it  can  be  saea  that  features  have  been  obtained  which  are 
proainent  in  soae  sense,  and  which  also  sitiafy  our  criteria  (to  a 
degree).  In  addition,  aost  texture  edges  are  filtered  out.  Third, 
areas  of  the  iaage  where  further  processing  night  be  nore  fruitful 
have  been  identified.  Fourth,  the  original  data  set  of  oriented  edges 
has  been  changed  and  reduced  to  a set  of  edge  segaents  which  contain 
information  of  a nore  global  nature. 

Finally,  the  edge  segaents  yield  areas  froa  which  correspondences 
between  two  images  can  be  generated,  based  on  soae  measure  of 
"saneness",  e.g.  correlation  of  segments  based  on  direction, 
location,  size,  etc.  Spatial  relationships  between  edge  segaents  will 
further  reduce  the  aaniguity  in  matching  of  features  in  two  iaages. 

This  prograa  has  created  features  by  linking  edges  along  angles 
to  fora  basically  straight  line  segaents  along  the  central  angle.  In 
figures  2b  and  2d  interangle  links  do  not  exist,  even  though  they 
appear  to  exist.  The  eye  creates  the  illusion  of  interangle  linking. 
This  first  step  has  considerably  reduced  the  nuaber  of  elements  to  be 
considered  for  further  processing.  The  next  step  is  to  link  the  edge 
segaents  from  different  angles  that  intersect.  Such  linking  will 
create  larger  edge  segaents.  The  threshold  for  nuaber  of  edge  points 
could  be  increased  to  eliminate  soas  smaller  segaents.  Thus  edges 
that  link  over  enougn  edge  points  over  one  angls  but  never  link  up  to 
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anything  else  would  be  eliminated.  This  will  result  in  further 
reduction  in  the  nuabec  of  features  without  eliminating  the  iaportant 
ones. 
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2.9  Clustering  for  laags  Segnentation 
Harry  C.  Andrews 

This  report  should  be  viewed  as  a broad  research  outline  for  an 
approach  to  autoaatic  iaage  segnentation.  The  technigue  is  based  upon 
methodologies  which  have  proven  successful  in  the  past  in  the 
coapanion  fields  of  pattern  recognition  and  signal  processing  of 
inages.  Specifically  the  approach  suggested  here  involves  a 
nathenatically  generalized  method  to  accept  features  or  attributes  of 
potential  segments,  to  computationally  select  the  best  reduced  set  of 
attributes  or  features  for  successful  segnentation,  and  then  to 
subsequently  attempt  hardware  sinplif ication  for  ultinate 
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implementation  at  the  sensor  end  of  an  ieage  acquisition  system.  The 
generalization  of  this  segsentation  sethod  sill  allow  subsequent 
addition  of  featurss  defined  by  new  operators  (as  they  are  invented) 
without  changing  the  sagmentation  technique.  This  then  will  allow  a 
quantitative  net hodo Logy  for  evaluating  new  attributes  as  they  cose 
along  while  correspondingly  improving  the  segsentation  results  without 
need  of  nodification  of  the  segaentation  procedure. 

Before  developing  tne  specifics  of  the  proposed  segsentation 
procedural  it  is  useful  to  describe  the  placeaent  of  this  technique 
within  the  heterarcaiil  formalism  of  0SC*s  approach  to  iaage 
understanding.  Essentially  the  iaage  segsentor  is  considered  a 
preprocessor  to  the  interpretation  element  of  the  inage  understanding 
which  initially  operates  as  a bottoa  up  processor  accusulating 
evidence  as  to  proper  segmentation  as  it  progresses.  This  is  not  to 
say  that  it  will  not  be  responsive  to  feedback  froa  the  higher  level 
world  nodels  and  interpretation  phases,  for  in  fact  this  feedback  will 
aid  in  the  optimization  of  the  reduction  of  necessary  features  for 
effective  segsentation. 

Finally  one  peripheral  application  of  a successful  iaage 
segaentation  procedure  consists  of  novel  iaage  coding  aethods.  These 
aethods  would  fall  into  the  category  of  "iaage  synthesis  via  iaage 
analysis,"  the  image  analysis  being  automatic  segaentation.  As  a 
simple  example  consider  a segment  which  has  a property  vector  in  N 
space  in  which  (for  simplicity)  each  property  coordinate  is  quantized 
to  5 bits.  This  implies  a total  of  32  different  textures,  32 
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intensities,  32  possible  means,  32  poteotial  variance  values,  etc. 

i 1 

| Thus  with  5 bits  per  property  and  I properties  per  sagnent,  it  nay  be 
possible  to  obtain  vary  large  bandvidth  reductions  through  cosbination 
of  the  attributes  of  3ejaents  with  segmentation  boundaries.  Clearly 
these  possibilities  need  to  be  investigated. 


I . 
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Property-attribute  Vector:  The  philosophy  of  "clustering  for 
inage  sagnentation"  is  to  dascribe  arbitrary  possibly  overlapping 
regions  of  an  iaage  by  a vector  of  attributes  or  properties  which 
huaans  feel  would  ns  useful  descriptors  to  differentiate  between 
segsents.  Because  of  the  lack  of  cosplete  understanding  of  how 
segmentation  occurs,  the  property  vector  will  probably  be 
overspecified  with  aigaly  redundant  attributes  or  coordinates. 
However  this  is  to  ba  expected  3ince  a "significant"  feature  selection 
operation  will  follow  the  property  vector  definition.  Before 
describing  the  feature  selection  and  dimensionality  reduction  aspects 
of  automatic  image  sagmantation,  it  will  be  useful  to  list  a few 
candidate  attributes: 
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local  area  aean 
local  area  variance 
local  area  histograns 
Robert's  operator 
Hueckel  operator 
Baker  operator 
other  edge  operators 

texture  operators  (joint  density  aeasures) 

entropy 

correlation 

intensities 

color#  R#G#B 

color  Y#  I#  Q 

color  L#C i #C2 

ratio  iaages 

angle  iaages 

depth  froa  stereo 

spatial  frequencies 

other  transfora  coefficients 

principal  coaponents 


The  above  list  is  not  aeant  to  be  all  inclusive#  but  is  provided  as  a 
set  of  suggested  first  alternatives  for  the  property  vector. 


Clustering  and  Feature  Selection:  The  N diaensional  vector 
describing  the  proparties  characterized  above  becoaes  a point  in 
(hopefully  Euclidean)  N space.  If  the  original  iaage  is  broken  into 
possibly  overlapping  snail  rectangular  sections#  and  if  each  section 
is  described  by  an  attribute  vector#  then  each  section  becoaes  a point 
in  H diaensional  space.  During  the  training  phase  of  the  clustering 
procedure#  a class  of  iaages  vill  be  selected  in  which  unaabiguous 
segaentation  exists  and  this  class  of  iaages  vill  be  sectioned# 
attributes  coaputed#  and  resulting  vectors  placed  as  points  in  ■ 
space.  Benenbering  that  the  axes  in  this  space  represent  textnre# 


color#  brightness#  etc.# 


pattern  recognition  clustering  procedures 
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will  than  be  applied  to  get  the  points  to  cluster  as  tightly  as 
possible.  Such  technigjes  include  convergence  algorithms  and  aeasures 
for  describing  tightness  of  clusters  using  between,  within,  and 
mixture  scatter  lattices.  Coupled  with  thase  will  be  feature 
selection  procedures  to  reiove  those  attributes  which  tend  to  scatter 
the  data  and  do  not  contribute  to  well  defined  clusters.  Typical 
aeasures  for  effecting  such  feature  selection  include  divergence  and 
Bbattacharyya  distances  as  useful  attributes  or  property 
discriainators  <1>. 


The  entire  procedure  of  sectioning,  property  vector  coaputation, 
iabedding  in  N dimensional  space,  clustering,  and  feature  selection  is 
directed  at  effective  aai  efficient  iaage  segmentation.  However,  it 
is  not  clear  what  a "properly  segaented  iaage"  is  or  what  a "better 
partition  of  the  iaage"  aeans.  Therefore  it  will  be  necessary  to 
direct  this  training  procedure  with  extra-image  knowledge,  i.e.  huaan 
intervention.  Much  was  accoaplished  in  the  way  of  the  developaent  of 
texture  discriainants  for  consistency  with  the  huaan  visual  process 
<2>,  a siailar  procedure  is  anticipated  in  the  definition  of  "correct 
partitioning"  in  the  sagaentation  problem.  Specifically,  imagery  will 
be  selected  in  which  huaan  interpretation  results  in  consistent  iaage 
segaentation.  A segmentation  discriainant  will  then  be  developed  inch 
as  in  the  texture  discriainant  work  <2>.  Extrapolation  to  aore 
difficult  imagery  will  follow  with  necessary  flam  tuning  of  the 
segaent  discriminator  via  additional  attributes  or  properties  and  sore 
precise  clustering  and  feature  selection. 
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2.10  Interpretation  and  World  Knowledge 
Erica  M . Rounds 


The  goal  of  thi3  research  is  to  design  the  decision-aaking 
coeponent  of  the  iaage-understanding  systee  which  will  construct  a 
aeaningful  description  of  the  real-world  scene  depicted  in  the  ieage. 
Research  efforts  are  concentrated  on  two  sub-problens: 

(a)  the  decision  procedures  which  assign  aaaning  to  the  regions 
or  parts  of  tne  ieage  discovered  during  the  segmentation  stage 
(interpretation*  ; and 

(b)  the  efficient  representation  of  objects  and  their  properties 
and  relations  in  the  probles  doaain  (knowledge  base) . 


Interpretation  is  essentially  a decision  process  designed  to 
select  froe  a set  of  possible  aeanings  one  which  fits  the  given  input 
inage.  There  is  an  obvious  sinilarity  to  the  classical  pattern 
recognition  problen,  the  nain  distinction  being  one  of  conplexity.  In 
inage  understanding,  the  physical  basis  of  the  inage  is  of  najor 
inportanca,  and  ths  inscriptions  of  object  classes  and  segnented 


Page  106 

regions  in  the  iaage  require  lore  coaplex  data  structures  than  is 
custoaary  in  pattern  recognition.  Howaver,  with  suitable 
nodifications,  soae  of  tne  pattern  recognition  aethodology  can  be 
usefully  applied  to  taa  problea  of  iaage  understanding  <1>. 

Decision  Process:  The  underlying  strategy  for  the  interpretation 
is  based  on  the  "hypothesis  foraation  and  validation"  paradiga.  This 
is  a two- phase  process  consisting  of  a long  range  strategy  (global 
search)  and  a short  range  strategy  (local  search) . Daring  tbe  long 
range  strategy,  a coarse  evaluation  of  the  regions  is  atteapted  to 
discover  soae  proaiaent  features  which  suggest  tentative  hypotheses 
concerning  tbe  objects  in  the  scene.  The  short  range  strategy  then 
accuaulates  evidence  for  tbe  hypotheses  by  parforaing  a aore  detailed 
analysis  of  the  pictorial  data.  This  paradiga  is  supported  by 
evidence  froa  huaan  visual  processing  which  suggest  that  an  initial 
overall  scanning  of  a scene  is  followed  by  a close  scrutiny  of  soae 
proainent  object  <2>. 


The  aodel  for  decision-asking  is  shown  in  figure  1.  Since 
interpretation  is  strongly  dependent  on  the  output  of  the  segnentation 
stage  and  conversely,  stgaentation  into  regions  nay  be  influenced  by 
partial  assignnent  of  regions,  the  segaentor  is  explicitly  included  in 
the  decision  nechanisn.  i full  Inage  understanding  systen  will 
require  tbe  integration  of  all  coaponents  froa  low-level  pictorial 
data  processing  to  higi-level  interpretation,  and  oust  provide  for 
aulti-directional  inforaation  flow  between  the  coaponents.  During  the 
initial  developaentai  stage  attention  will  be  directed  to  the 


Figure  2.  10-1.  Decision  Mechanism, 


Page  108 


interaction  between  the  interpretation  and  segaantation  nodules. 

The  global  search  requires  selection  criteria  for  choosing  a 
candidate  aodel  froa  a set  of  possible  aodels.  In  addition, 
provisions  for  backtracking  oust  be  included,  in  case  the  local  natch 
fails.  Backtracking  should  avoid  redundant  work  by  eliainating  entire 
subtrees  of  the  search  tree  as  soon  as  possible,  A convenient  way  for 
describing  sequential  decision-aaking  is  by  aeans  of  a decision  tree 
as  shown  in  figure  2.  The  square  nodes  are  decision  nodes  at  which 
selection  or  teraination  criteria  are  applied,  and  the  circular  nodes 
are  chance  nodes  whosa  successors  are  deternined  by  the  possible 
values  of  the  tested  features.  The  tern  feature  as  used  here  refers 
to  attributes  of  objects  and  to  relations  between  objects  and  between 
conponents  of  objects.  This  type  of  decision  tree  closely  reseables 
the  so-called  AND-OH  trees  in  gaae  playing,  where  tha  objective  is  to 
optiaize  the  strategy  for  one  player  (square)  issuaing  that  the  other 
player  (circle)  always  responds  with  the  best  possible  aove. 

At  aach  decision  node  three  alternative  decisions  are  possible: 


(a)  continue  with  current  hypothesis  and  salect  a new  feature  to 
be  tested  whicn  will  contribute  aaziailly  to  the  accuaulated 
evidence; 

(b)  discard  the  current  hypothesis  and  selact  a new  aodel 
(backtrack);  and 

(c)  accept  the  cicrent  hypothesis  as  the  final  interpretation  of 
the  scene  (teraiaate  search). 


Selection  and  tarnination  criteria  are  based  on  aoae  quantitative 
aeasnce  defined  on  the  attributes  and  relations.  In  statistical 
pattern  recognition,  probability  distributions  aro  assigned  to  pattern 
classes  and  features.  Given  the  great  coaplezity  and  variety  of 
objects  in  real-world  scenes  it  is  doubtful  if  one  can  extract 
reliable  probabilty  estinates  froa  saaple  scenes  <3>.  The  plan  is  to 
associate  a weight  or  index  of  significance  with  features.  The 
weights  nay  be  considered  as  a refinenent  of  Boolean  branching  logic, 
or  as  a coarse  estiaate  of  expected  occurrences  of  relations  and 
attributes  with  respect  to  given  object  classes.  Heights  will  be  used 
priaarily  as  a heuristic  device  to  reduce  the  search  effort. 

Knowledge  Representation;  Different  levels  of  processing  require 
different  kinds  of  knowledge.  The  entire  process  of  iaage 
understanding  can  be  viewed  as  successive  levels  of  abstraction,  i.e. 
froa  a aass  of  raw  input  data  to  a single  synbolic  description  of  the 

v 

scene.  The  anount  of  detail  and  the  structure  of  the  description  will 
depend  to  a significant  extent  on  the  goal  of  the  systea.  For 
exaaple,  one  nay  wisi  to  locate  a particular  object  in  the  scene  as 
done  in  <3>,  or  produce  an  overall  description  of  the  scene,  or  one 
nay  wish  to  extract  enough  inforaation  so  that  the  systea  can  answer 
neaningful  questions  about  the  scene. 

The  systen  is  to  be  applied  to  twenty  iaages  of  outdoor  scenes 
with  a coabination  of  aan-aade  and  natural  objects.  The  first  problea 
donain  i3  derived  froa  aerial  photographs  shoving  various  vehicles  in 
a desert-like  environaent.  Object  classes  consist  of  vegetation. 
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bushes,  sand,  tracks,  aad  vehicles  of  type  tank,  truck  and  airplane. 
The  goal  of  the  systaa  is  to  detect  and  interpret  tha  aan-aade  objects 
froa  the  background  iasart  and  vegetation.  The  interpretation  module 
Hill  construct  a conceptual  eodal  of  an  input  iaage  including  a 
description  of  eajor  objects  and  structural  information  relating  these 
objects  to  the  scene  as  a whole. 

As  indicated  ia  figure  1,  knowledge  used  by  the  decision 
processes  is  roughly  divided  into  visual  data,  region  descriptions 
produced  by  the  segaantoc,  and  knowledge  about  tha  problem  domain 
retrieved  froa  the  knowledge  base.  The  first  two  kinds  of  information 
change  dynamically  during  processing  and  may  be  considered  part  of  the 
short-term  memory  weich  stores  the  iaage  descriptions  produced  at 
different  levels.  Tha  knowledge  base  is  static  and  constitutes  the 
long-term  memory  containing  models  of  scenes  and  objects. 


In  reference  <1>  decision  trees  provide  an  implicit  description 
of  the  object  classes.  A path  through  the  tree  specifies  which 
features  a region  or  set  of  regions  must  possess  in  order  to  be 
assigned  to  a gives  ooject  class.  In  our  system  object  descriptions 
are  explicitly  stored  in  the  knowledge  base.  This  has  the  advantage 


that  the  decision  processes  are  largely  independent  of  the  problem 
domain.  Thus  the  systea  can  achieve  a certain  degree  of  generality 
and  may  be  applied  with  relatively  few  modifications  to  other  problem 
domains  by  providing  a different  knowledge  base.  Similar 
considerations  of  flaxioility  have  influenced  the  choice  of  a two-part 
search.  The  long-range  strategy  uses  a global  decision  tree  to  select 
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a aodel  on  the  basis  of  a few  discriMinatory  features.  The  local 
search  then  attempts  a latching  of  region  descriptions  against  Model 
descriptions  in  the  knowledge  base.  This  approach  can  easily  be 
extended  to  the  case  where  input  iiages  aay  come  for  a number  of 
different  problea  doaains.  The  global  strategy  would  first  select  the 
applicable  problea  doaain  followed  by  a selection  of  a candidate 
interpretation. 

A aajor  problaa  in  developing  the  knowledge  base  is  the 
identification  of  ralavant  features.  Criteria  for  "good"  features 
include  discriainator y power  and  a certain  degree  of  invariance,  e.g. 
with  respect  to  viewing  angle  and  scale.  In  addition,  seaantic 
knowledge  such  as  noapictorial  attributes  eay  be  essential  for 
unaabiguous  interpretation  <5>.  Under  consideration  are  the  following 
features  which  have  proved  useful  at  the  Interpretation  level: 

(a)  optical  properties  (texture,  shadows,  highlights, 

reflectivity) 

(b)  geometric  attributes  (relative  size,  location,  shape) 

(c)  spatial  relations  (distance,  sixe  ratios,  orientation) 

(d)  topological  relations  (adjacency,  containment,  occlusion, 

connectivity) 

(e)  semantic  information  (functional,  spatial  and  temporal 

context) 

Initially,  scene  and  object  descriptions  will  be  derived  by 
segmenting  the  digitize!  images  of  the  example  problea  doaain  manually 
and  selecting  features  using  human  protocol.  To  reconstruct  objects 


seen  froa  different  perspectives,  objects  descriptions  will  be  stored 
in  three  canonical  views:  front,  top,  and  side.  Representation  of 
world  knowledge  is  in  the  fora  of  graph-like  structures  similar  to 
semantic  nets,  where  the  nodes  represent  concepts,  object  categories, 
objects,  and  components  of  objects,  and  the  edges  represent  relations 
between  nodes  or  properties  <6>.  The  weights  discussed  above  are 
associated  with  the  edges.  The  grouping  of  objects  into  categories 
which  share  common  properties  allows  a more  parsimonious  description 
and  will  reduce  the  search  effort.  The  major  categories  are  natural 
objects  and  man-made  objects  which  are  further  subdivided  by  size, 
shape,  texture,  etc. 

Too  much  specialized  detail  stored  in  the  knowledge  base  will 
impede  matching  with  ragion  descriptions.  The  system  must  be  able  to 
tolerate  deviations  froa  the  expected  prototypes.  Our  approach  is 
related  to  Minsky's  frame  concept  <7>,  i.e.  certain  features  are 
labelled  as  essential,  which  must  always  be  matched,  and  others  are 
nonessential  which  lay  be  assigned  a default  value  representing  the 
most  likely  value. 

The  foregoing  discassion  has  provided  an  overview  of  the  approach 
in  the  design  of  an  interpretation  nodule.  Many  details  have  not  been 
covered  because  the  ideas  are  not  yet  fully  developed.  The 
interpretation  nodule,  for  example,  will  consist  of  sany  other 
subprocesses  in  addition  to  the  hypothesis  generator  and  aatcbing 
process.  Region  descriptions  obtained  froa  the  segnentor  must  be 
analyzed  for  shape,  shadows  and  related  properties.  Modular 
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dovelopaent  of  the  systea  will  facilitate  the  addition  of  new 

I 

subprocesses. 
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3.  Iaage  Processing  Projects 

The  isage  processing  projects  coaprise  an  ongoing  research 
activity  directed  tovard  iaage  coding,  iaage  restoration,  vision 
aodellin g,  and  the  iaplsaentation  of  iaage  processing  systeas.  In 
iaage  coding  novel  ideas  based  upon  the  results  of  the  iaage 
understanding  study  are  being  explored  as  a aeans  of  achieving 
significantly  highar  coapression  ratios  than  obtainable  by 
conventional  coding  aathods.  The  iaage  restoration  studies  are 
directed  toward  the  solution  of  two  aajor  probleas:  blind  restoration 
in  which  a priori  inforaation  about  iaage  degradation  is  unavailable 
or  incoaplete;  and  constrained  restoration  which  involves  the  use  of 
luainance  bounds  and  saoothness  criteria  to  inproved  iaage 
restoration.  Vision  Modelling  research  activities  include  the 
extension  of  previously  developed  aodels  of  the  huaan  visual  systea  to 
encoapass  higher  lavaLs  of  visual  perception  in  support  of  the  iaage 
understanding  prograa.  Iapleaentation  studies  are  underway  on 
techniques  of  nonlinear,  two  diaensional  optical  filtering  which  can 
be  utilized  as  a fora  of  sensor  based  iaage  processing. 

3.1  Snoothing  Splines  for  Spatially  Variable  Restoration 
H.  Javad  Peyrovian  and  Alexander  A.  Sawchuk 

As  a continuation  of  previous  work  <1>,  spline  functions  have 
been  utilized  for  locally  variable  filtering  restoration  of  noisy 
blurred  iaages  <8> . For  an  unblurrad  noisy  iaage  the  iaage  g(x)  is 
given  by 
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g(x)  = f(x)  + n(x) 


(3. 1-1) 


The  restoration  operation  is  designed  to  ainiaize  the  integral 

fi  f'(x)]2dx  ’ (3.1-2) 

anong  all  twice  diffa raatiable  Cnnctions  f (z)  such  that 


(3. 1-3) 


where  6 locally  controls  a saoothing  window  at  point  and  S controls 
the  overall  extent  of  snoothing*  If  available,  a the  standard 

deviation  of  the  noisa  at  point  x.  can  be  used  for  6.  . in  this 

i i 

case,  natural  values  of  S lie  within  the  confidence  interval  of  the 
left  hand  side  of  eg.  (3)  That  is 


N - (2N)*  * S * N + (2N)*  (3. 1-4) 

where  N is  the  nuaber  of  data  points.  Reinsch  <2>  has  shown  that  the 
solution  to  egs. (2)  and  (3)  is  a cubic  spline  and  wore  generally  is  a 
spline  function  of  degree  2K-1  for  least  sguare  ainiaization  of  the 
K th  derivative  instead  of  the  second  derivative.  The  case  K - 2 
leads  to  very  siaple  algorithms  for  the  construction  of  the  function 

f <*)  • 


Assuming  f 

to  be  a 

polynoaial  of 

degree 

3 

or  less  in 

each 

interval. 

the 

optiaal 

restoration 

filter 

with 

respect  to 

the 

conditions 

of 

egs.  (2) 

and  (3)  can 

be  obtained 

by  calculus 

of 

variations  <2>.  The  solution  is 
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2 T 2 -IT 

i = 8 - £>  Q (Q  D Q + pT ) Q g 

where 


(3. l-5a) 


1 =Cf(*1),  £(3^) ff^)]4 

(3.  l-5b) 

£ ■ ^ 8 (*2),  • * • , b(x^)] 

(3.  l-5c) 

D = Diag  Cftj,  62,...  *N] 

(3.  l-5d) 

Matrix  T is  a positive- definite  tridiagonal  aatrix  of  order  M-2,  and  2 
is  a tridiagonal  aatrix  with  N rows  and  N-2  ooluans.  For  a uniforaly 
saapled  iaage,  Q and  ^ have  the  fora 

i 1 


(3. 1-6) 


T • 


4/3  1/3 

l/3„  4/3^  l/3_ 


1/3  . 4/3 
1/3 


1/3 

4/3 


(3.  1-7) 


where  p is  a Lagrangiau  paraaetar  which,  like  S,  controls  the  overall 
extent  of  saoothinj.  Since  _T  is  invertible,  by  using  aatrix 
identities  <3>,  eg.  (5)  can  be  written  in  the  aore  concise  fora 

f=  (I+P-1  P2QT"1QT)"1  g (3.1-8) 


Although  eg.  (8)  appears  siapler,  it  needs  two  aatrix  inversions  in 
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comparison  to  eg.  (5)  which  requires  the  inverse  of  a banded  eatrix. 

T 

The  CholesAy  decoaposition  <4>  8 I of  a positive-definite  band  aatrix 

T 2 

Q D Q ♦ pT,  where  B is  a lower  diagonal  (triangular)  matrix,  provides 
an  efficient  computational  algorithm. 

in  interesting  property  of  the  fidelity  criterion  of  eg.  (3)  is 
that  the  smoothing  window  can  be  locally  controlled  by  determination 
of  6.  . It  the  noise  variance  is  higher  in  some  regions,  6.  can  be  set 
larger  at  that  region.  An  application  of  this  property  is  the 
filtering  of  images  with  film-grain  noise,  modeled  <5>  by 


D = D 

r a 


+ D* 


(3.1-9) 


where  Ds  is  the  signal  density,  DT  is  recorded  density  and  n is  zero 
mean  noise.  Eguation  (9)  shows  that  when  the  signal  has  a higher 
amplitude,  nois*  has  a greater  variance.  To  apply  the  above  mentioned 
filter,  the  value  of  6.  and  an  estimate  of  the  signal  are  needed. 
Hunt  <6>  has  shown  that  images  are  well  described  statistically  as  a 
stationary  variance  about  a spatially  non-stationary  local  mean. 
Assuming  ergodicity  of  similar  classes  of  images,  the  spatial  average 
aay  be  used  as  an  estiaate  of  the  ensemble  average.  A local  spatial 
average  is  thus  used  as  a non-stationary  estiaate  of  the  signal  mean. 


A noisy  blurred  iaage  can  be  expressed  in  the  form 


g = HI  + n 


(3.1-10) 


for  a discrete  model.  The  following  fidelity  criteria  for  iaage 
restoration  nay  then  be  formulated  as  minimization  of 
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f (f ' (*))J 


(3.1-11) 


among  all  f (x)  such  that 


HD  (H£-f)  II2  s S 


where  D,  t and  ^ are  previously  defined.  Using  the  saee  procedure  as 


before*  the  estisate 


* T -2  -1  T -1  T -2 

f *(H  D H + XQT  Qa)  aH  D g 


is  obtained*  where  X * 1/p.  Setting  B * l leads  to  the  estisate  of 
eg.  (8)  . Comparison  of  this  restoration  filter  to  the  Wiener  filter 
aatriz 

W=  (HTC"1  H + C"1)"1  H TC-1  (• 

— v—  — n — —I  — — n 

-1  -1  T 

shows  the  similarity  of  these  two  filters  with  replaced  by  X£T_ £ . 

In  order  to  obtain  the  filter  numerically*  it  is  necessary  to  invert 

T -2  -IT 

matrix  T as  well  as  H.  £ • Batrix  T is  a positive-definite 

tridiagonal  matrix;  efficient  technigues  exist  for  computation  of  its 
inverse  <7>.  Matrix  Us  a rectangular  H x N matrix  where  H < N for 
non-zero  background  pictures*  thus  an  exact  inverse  does  not  exist  for 
ttT£"\*XQT” \}T,  and  the  pseudo-inverse  must  be  employed. 

The  filter  of  eg.  (5)  has  been  applied  to  an  image  corrupted  by 

signal-dependent  noise  modelled  by  eg. (9) . Figure  la  is  the  original 

picture.  Figure  Id  is  a plot  of  the  brigatnass  cross-section  of 

figure  la,  and  the  subsaguent  pictures  are  noisy  and  filtered  images. 

2 

The  noise  is  Gaussian  with  variance  a = US  where  S is  the  signal 


Figure  3.  1-1.  Filtering  of  signal-dependent  noisy  images 
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intensity.  The  filtar  has  reduced  the  aean  square  error  by  a factor 

of  tea.  The  filter  of  eg.  (13)  ia  applied  to  a blurred  aoiay  iaage  as 

shown  ia  figure  2 where  the  point  spread  function  is  unifors  (notion 

degradation)  and  the  noise  is  Gaussian  with  standard  deviations  of  1 

and  5.  The  observed  iaage  has  a nonzero  background#  and  therefore# 

the  systea  is  undeteraiaed.  The  seoothing  pnraneter  X is  set  to  0.05 

and  1.0  and  the  images  are  restored  using  thresholds  e « 0.001  and 

T -2  -IT 

e » 0.01  for  the  singular  values  of  the  eatrix  JI  £ • is  e 

decreases,  the  restored  iaage  becoaes  sharper#  but  at  the  sane  tine 
the  aaplitude  of  unwanted  high  freguency  components  increases.  The 
quality  -of  the  restored  iaage  is  highly  dependent  on  the  proper 
selection  of  X and  e . 

This  work  has  deaonstrated  the  usefulness  of  spline  functions  for 
digital  iaage  processing.  in  optisal  filtar  with  respect  to  the 
second  derivative  has  been  derived  and  applied  to  noisy  and  blurred 
iaages.  although  the  experimental  results  are  prelininary#  they  show 
that  spline  functions  are  very  useful  in  digital  filtering  and 
restoration.  Moreover#  since  all  the  natrices  are  banded#  these 
filters  are  computationally  very  efficient. 
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3.2  Least  Sguares  Iaage  Restoration  Osing  Spline  Basis  Functions 
Steve  Hou  and  Barry  C.  Andrews 

This  contribution  presents  a theoretical  analysis  and 
cosputational  technigue  for  constrained  least  sguares  inage 
restoration  using  spline  basis  functions.  A realistic 
continuous-discrete  physical  inaging  nodel  has  been  adopted  throughout 
the  fornulation.  The  optical  systes  is  assuaed  to  be  incoherent,  and 
the  general  problea  of  isage  restoration  with  space-variant  or 
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ipuce-inraria nt  point-spread  function  degradations  has  been  studied. 

A noraal  equation  aas  been  formulated  in  a finite  dieensional 
spline  space.  The  aagnitude  of  the  saoothing  paraneter  in  the  noraal 
equation  provides  freedoa  to  control  resolution  in  a trade-off  of 
saoothing  the  restored  object.  This  effect  has  been  ieaonstrated  with 
experimental  results.  Constraints  that  characterize  the  physical 
properties  of  the  restored  object  have  been  formulated  so  that  they 
can  be  iaposed  on  the  solution  of  the  noraal  equation.  These 
constraints  require  the  restored  object  pizels  be  positive  and  the 
energy  of  the  object  be  equal  to  that  of  the  degraded  image. 

Iterative  methods  for  both  unconstrained  and  constrained 
solutions  of  the  noraal  equation  have  been  studied  in  detail.  Anong 
the  unconstrained  aetaods  the  conjugate  gradient  eethod  has  been 
successfully  sieulatad  on  a coaputer  and  strikingly  good  results  have 
been  obtained. 

The  needs  of  digital  inage  restoration  arise  fron  the  fact  that 
an  iaage  in  the  real  world  is  either  corrupted  by  noise  or  degraded  by 
various  physical  phenoaana  during  formation,  such  as  the  diffraction 
liait  of  the  optical  systea,  sensor  and  display  non-linearities, 
optical  systea  aberrations,  ataospheric  turbulence,  iaage  notion  blur, 
geoeetrical  distortion  and  file  grain  or  sensor  noise. 

The  aatheaatical  approach  consists  of  finding  a function  f such 
that  a known  operator  transforas  f into  a given  function  g.  In  the 

A 

context  of  iaage  restoration,  an  estiaate  f of  the  object  is  sought  to 


Page  125 


be  as  close  as  passible  to  the  ileal  f that  produced  g.  There  are 
aany  difficulties  associated  with  direct  siuiaizatioa  of  ||f-f||.  First 
the  inverse  of  the  operator  nay  not  exist  or  even  if  it  exists,  it  is 
jsually  very  hard  to  fiad.  Second  in  aost  physical  iaaging  systeas 
the  ideal  object  loes  not  exist  in  a aeasurable  fora,  but  siaply  is 
the  conceptual  object  that  would  be  producel  if  there  were  no 
degradation  and  noise  present  in  the  iaaging  systea.  Third  g is  only 
partially  known  because  it  results  froa  aeasureaents.  To  avoid  these 
difficulties,  a certain  fiielity  criterion  of  iaage  quality  is 
aatheaatically  defined  ia  order  to  judge  the  degree  of  iaage 
restoration.  The  iaage  restoration  aodel  is  based  upon  continuous 
object-discrete  iaage  description  and  a Predhola  integral  in  two 


variables  given  by 


G = f f Hf?.T|)f<S.Tl)dSdTl  + N 


(3.2-1) 


where  6 is  an  I x J aatrix  with  siasaat  g^  , £ is  an  I x <1  aatrix  with 
elesent  h.j  (? ,ri)  and  £ is  also  an  I x J aatrix  with  aoise  saaples  n.j 
as  its  eleaents.  This  aodel  describes  an  object  scene  as  a continuous 
function  in  two  disaasional  space,  as  it  should  be  in  the  real  world, 
but  the  iaage  at  the  3ensor  output  (or  at  the  input  of  a processing 
coaputer)  is  discrete,  and  has  been  stapled  into  I x J points.  This 
is  certainly  a realistic  aodel  for  digital  iaage  restoration  systeas, 
and  will  be  pursued  ia  detail  throughout  this  report.  Note  that  the 
aatrix  H(?,ti)  is  a point  spread  function  aatrix  whose  entries  are 
continuous  functions  of  the  object  plana  (S,t|)  ns  well  as  the  sanpling 
position  in  the  iaaga  plane  (i,j).  Clearly  this  point  spread  aatrix 
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in  its  aost  general  fora  can  be  space-variant. 

To  proceed  it  is  necessary  to  establish  certain  criterion  on 
iaage  quality  for  rsstoration  that  peraits  control  of  the  resolution 
versus  noise  saoothing  on  the  final  restored  oDject.  The  criterion 
used  is  a aodified  least  sguares  objective  function.  By  ainiaizing 
this  objective  function  it  is  possible  to  foraulate  a linear# 
algebraic  equation  for  solving  the  unknown  paranetars.  However  these 
unknowns  oust  be  discrete  and  have  finite  uuabsrs#  since  a digital 
coaputer  can  only  nandle  a liaitel  aaount  of  discrete  data.  On  the 
other  hand  an  estiaation  is  sought  for  a continuous  object  which  needs 
infinite  data  points  for  its  description.  Hence  an  interpolation 
scheae  aust  be  design  ad  to  restore  the  continuous  object  froa  a finite 
set  of  discrete  data.  Froa  a data  coapression  point  of  view#  this 
interpolation  approaca  is  attractive  because  it  is  only  necessary  to 
solve  a linear#  algebraic  equation  in  finite-diaensional  space  instead 
of  the  integral  equation  of  eq.(1). 

Pictorial  iaage  processing  usually  requires  handling  a large 
anount  of  data.  Effective  iterative  Methods  aust  be  considered  for 
the  solution  of  the  Linear#  algebraic  equations  derived  froa  the 
ainiaization  of  the  objactive  function.  This  iterative  algorithn  aust 
also  be  tested  for  its  speed  of  convergence,  for  its  usefulness  for 
iaage  restoration. 

In  the  foraulation  of  the  equations  for  iaage  restoration#  three 
points  aust  be  considered:  (a)#  tradeoffs  between  iaage  resolution  and 
aoise  snoothing;  (b) # feasibility  of  auaerical  calculations  of  very 
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large  size  satcices;  (c) , constraints  isposel  on  the  equations  by  a 
priori  knowledge  about  the  object  function.  To  satisfy  the  first 
point  the  objective  function  consisting  of  the  sue  of  the  noise  nors 
in  eq. (1)  and  a snoot bin 3 integral  is  chosen  for  sinieization.  Por 
the  second  point,  a f inite-diaensional  space  is  chosen  in  which  the 

A 

objoct  function  f is  a linear  cosbination  of  the  cubic  B-spline 
functions,  i.e.,  the  bases  are  cubic  B~splines.  This  choice  enables 
conversion  of  an  infinite~diaensional  or  continuous  probles  such  as 
that  in  eq.  (1)  into  a discrete  one,  to  obtain  computationally 
attractive  solutions.  Por  the  third  point,  bounded,  lossless  iaaging 
restoration  aentional  above,  is  utilized.  The  positive  restoration 
and  lossless  iaaging  assumptions  add  additional  constraints  to  the 
solution  although  tae  solution  is  still  not  unique  because  of 
ill-conditioni ng. 


To  derive  a funlaaantal  equation  for  image  restoration  in  two 
dimensions,  the  so  called  normal  equation  from  the  least  square 

A 

criterion,  an  objective  function,  which  is  minimum  at  f(?,T|),  is 
defined  as 


where 


W(f)  = Tr[(G-G)  (G-G)]  + Y 


[V4f(?,n)]2d?dn 


Hf(?,r1)d§dr1 


(3.  2-2) 

(3.  2-3a) 
(3.  2-3b) 
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a ad 


2 -f  f?m  , IT)  d§  dri 


(3. 2-3c) 


is  a aatrix  that  has  tha  saae  size  as  G which  is  I x J.  The  elements 


, foe  i = 1 to  I and  j = 1 to  J,  of  aatrix  G are  the  seasured  data. 


ji  is  tha  point  spread  aatrix,  with  elements  h _ (?,ri).  The  second  ters 
iu  eq.  (2)  is  the  desired  saoothing  aeasure  for  the  class  of  estiaated 

A A 

objects,  f(?,Ti).  The  function  f(?/n)  can  be  linearly  interpolated  by 

a product  of  two  one- diaensional  cubic  B-aplinas.  Hence  let 

K L 

= £ 13  Ck<.'k,?,v,'l)  <3- 2-4) 

k= 1 1=1 
k=l  1=1 

where  ( ?)  and  s^  (td  are  one-diaensional  cubic  B-aplines  and  \t  is 
the  product  of  s with  s . 

K i/ 

* - 

The  next  step  is  to  find  an  fjS/n)  i.e. , a particular  set  of 

2 A 
coefficients  {c^}  in  eg.  (4)  such  that  the  objective  function  W (f ) in 

aq.  (2)  attains  its  liniaua.  This  can  be  achieved  by  solving  the 

noraal  equation 


K.iS.'n) 


(A  + YB)  c = d 


(3.2-5) 


w here 


00  ep 

J J «(?,Ti)TrCHT(?,n)ii(?'n«)]lT? 

B =ff  7 4 i(5.  n ) ?/(§.  ri)  d?dri 


T)1  )d?  dT|  d^’  dr)1 


(3. 2-6a) 


(3.  2-6b) 
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T|)  Tr[H  (?,  Ti)G]d5dTi 


Equation  (5)  deserves  special  attention  because  of  its  general 
Cora*  It  should  be  observed  that  eg.  (5)  can  be  further  reduced  to  the 


fora 


where 


— T~  ~T 

(A  A + YB^  ® By)  c = A g 


-(//  tij(?'T1)8k(§)8f!T1)d§dT1j 

-v(£ 

-b AL 


X KL 


and  (£)  denotes  the  Kronecfcer  (direct  or  teaaor)  product. 


(3.2-7) 


(3.  2-8a) 


(3.  2-8b) 


(3. 2-8c) 


The  following  aataeaatical  properties  exist  for  the  aatrices  A 


and  B: 


(a)  A.  and  B are  both  real,  syaaetric. 

(b)  a > a and  b > b for  p 4 a. 

pp  pq  PP  pq  * * 

(c)  k is  non-negative  definite  with  all  positive  eleaents. 

(d)  J is  positive  Infinite. 

In  addition,  froa  egs. (3a)  and  (8a)  it  is  clear  that  k is  essentially 
the  anount  of  correlation  or  overlap  of  the  point  spread  function 
represents}  in  splina  space.  Thus  the  effect  of  the  blur  degradation 
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T 

aaoifests  itself  in  the  rank  or  degree  of  A (or  X . Thus  1 behaves 
as  a Graaiaa  describing  the  isaging  aodel,  but  in  spline  space.  Hence 
A is  non-negative'  definite.  The  elesents  a and  a are  always 

- pq  pq 

non-negative  because  of  the  non-negativeness  of  h..  (5,n),  i (?,r|)  and 

P 

$ The  value  of  the  eleaents  of  A in  eg.  (8)  depends  on  the  PSP 

q — 

overlap  area.  For  finite  extended  h„  (?,ri)  sparseness  of  aatrix  A can 
be  expected.  Matrix  B represents  the  correlation overlap  of  the 
second  derivative  of  the  spline  basis  functions  with  thenselves.  As  a 
result  of  the  separability  of  the  basis  functions,  B can  be 
represented  as  a kronecter  product  as  in  eg. (7) . Again  B represent 
the  "Graaian"  of  toe  second  derivative  basis  functions,  but  is 
deterainistically  known  because  of  the  expansion  and  interpolation  on 
cubic  B-spline  space.  This  results  in  a banded  structure.  The 
aatrices  and  are  non-negative  definite  banded  aatrices. 

Unconstrained  Restoration:  The  foraulation  of  AfyB  in  eg.  (5)  is 
often  referred  to  as  the  Tikhonov  regularization  aethod  <1>.  Much 
Russian  literature  <2-5>  has  bean  devoted  to  the  special  case  of 
Tat  none  has  presented  a coaplete  solution  for  the  general 
case.  A sinple  ninded  approach  for  arriving  at  a solution  is  to 
invert  A*YB  directly.  Jowever  there  are  two  reasons  for  avoiding  this 
direct  approach.  One  reason  is  that  the  aatrix  A+YB  is  usually  very 
large  in  iaage  processing  practice.  Its  inversion  aay  reguire  a huge 

neaory  space,  which  is  just  not  available  even  in  aodern  computers. 

-vT 

The  other  reason  is  that  the  aatrix  l * I I is  freguently 
ill-conditioned  and  influenced  greatly  by  roundoff  errors.  A 
nuaerical  exaaple  of  this  kind  has  been  shown  by  Golub  <6>.  Adding  a 
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positive  definite  aatrix  yB_ to  k nay  lessen  this  difficulty,  but  still 
not  completely  eliminate  it  because  .B  is  an  alaost  periodic  aatrix. 

In  this  report  aa  iterative  technique  provided  by  the  conjugate 
gradient  aethod  is  considered.  The  advantages  of  adopting  an 
iterative  approach  are  the  ability  to  aonitor  singularity  developing 
during  iteration,  aad  perait  human  participation  in  the  convergence 
process.  The  conjugate  gradient  aethod  is  useful  for  the  solution  of 
a systea  of  siaultaneous  linear  equations  with  an  arbitrary  syaaetric 
positive  definite  aatrix.  It  is  especially  advantageous  if  the  aatrix 
is  not  full  but  contains  aany  zero  elements.  In  the  problen  as  posed 
by  eg.  (5),  the  aatrix  A* y IB  is  syaaetric  positive  definite  if  y is 
positive  and  non-zero.  Matrix  _A  is  sparse  if  the  point  spread 

function  is  localized;  xatrix  B is  always  sparse.  In  the  case  Y = 0 
and  A is  singular,  the  conjugate  gradient  aathod  will  give  a least 
squares  solution. 

As  an  introduction  to  the  gradient  aethod  consider  the  quadratic 

fora 

J(£)  = i cT(A  ♦ Yb)c  - cTd  (3.2-9] 

vith  the  assuaption  taat  (A+yB)  is  positive  definite  and  syaaetric. 
Taking  the  partial  derivative  of  J (c)  vith  respect  to  c gives 

Grad  J(c)  « r = (4  + Yj})c  - d ( 

Hence  the  solution  of  a syaaetric  positive  definite  systea  of 
equations  of  the  fora  (A+YB)c  * d is  equivalent  to  the  problea  of 


finding  the  ainiaua  of  a quadratic  function  of  eq. (9) . Por  this 
reason  the  gradient  vactor  r in  eq.(10)  is  called  the  residual  vector. 


Gaoaatrically , tie  contour  surfaces  of  constant  J (c)  represent 
concentric  ellipsoids.  The  gradient  _r  at  every  _c  is  noraal  to  the 
surface  at  that  point.  Finding  the  ainiaua  of  J (£)  oy  varying  c is 
equivalent  to  aovinj  the  vector  c in  a certain  direction  for  finding 
the  coaaon  center  of  the  faaily  of  ellipsoids  of  J(c>  * constant. 

An  iterative  iapleaeutation  requiring  no  aatrix  inversions  has 
been  discovered.  The  conjugate  gradient  algoritha  offers  the 
following  additional  advantages  <7,8>. 

(a)  Siaplicity  of  the  coaputational  procedures. 

(b)  The  need  for  only  a Halted  aaount  of  storage  space  as 

coapared  with  the  variable  aetric  aethod  <9,10>. 

(c)  The  preservation  of  the  original  aatrices  of  coefficients 

such  as  k,  B , B , and  G during  tha  coaputation  and  the 
~x  y 

consequent  ability  to  exploit  the  advantages  of  having  aany  xero 

eleaents  in  these  aatrices. 

(d)  The  ability  to  start  anew  at  any  point  in  the  coaputation. 

(e)  The  superiority  of  each  approxiaation  to  tha  preceding  ones 

in  the  sense  of  being  closer  to  the  true  solution. 

Por  a aore  extensive  description  of  this  technique,  coaplete 
details  are  available  in  OSCIPI  Report  650,  (larch  1976.  In  addition 
experiaental  results  as  well  as  spline  interpolation  and  constrained 
optiaization  are  incljded  in  that  report. 


1 


2.  A.N.  Tichonov,  " 9 egularization  of  Incorrectly  Posed  Problems,” 
Soviet  Nath.  Dokl.,  Voi.  4,  1963,  pp.  1624-1627. 


3.  I. I. 

Khudak,  "On 

tae  Convergence 

of 

Regularizing 

Algoritha,"  Zh. 

V ychial. 

Nat.  aat. 

Piz.,  Vol.  11, 

No. 

1 r 1371,  pp 

. 29-35. 

4.  Y.I. 

Khudak,  "On 

the  Convergence 

of 

a Paaily 

of  Regularizing 

Algorithas,"  Zh.  Vychial.  Nat.  ait.  Piz. , Vol.  12,  No.  2,  1972, 
pp.  497-502. 

5.  S.D.  Dreizin-D uichenko  and  M.K.  Lilcht,  "Approxiaation  by 
Polygonal  Paths  in  a Quadratic  Metric  in  the  Solution  of  Integral 
Equations  of  the  First  Kind,"  Zh,  Vychial.  Hat.  aat.  Fix.,  Vol. 
11,  No.  5,  1971,  pp.  1301-1306. 

6.  G.H.  Golub,  "Matrix  Decoaposition  and  Statistical  Calculations," 
in  Statistical  Coaputation,  edited  by  B.C.  Hilton  and  J.l.  N elder, 
Acadeaic  Press,  New  Tort,  New  York,  1969,  pp.  365-367. 

7.  N.B.  Hestenes,  "The  Conjugate  Gradient  Method  for  Solving  Linear 
Systeas,"  Proceedings  of  Syaposina  in  Applied  Nathenatics  and 
Nunerical  Analysis,  1956,  pp.  83-102. 


Page  134 


0.  P.S.  Beckman,  "The  Solution  of  Linear  Bguations  by  the  Conjugate 
Sradieut  Method,"  in  Mathematical  Methods  foe  Digital  Computers,  Vol. 

1,  John  Wiley,  1967,  pp.  62-72. 

9.  W.C.  Davidon,  "Variable  Algorithm  for  Minimization,"  Computer  J. , 

Vol.  10,  1968,  pp.  405-410. 

10.  W.C.  Davidon,  "Variable  Metric  Nethol  for  Minimization," 
ANL-5990'  ABC  Research  Development  Report,  1959. 

3.3  Past  Sequential  SVD  Pseudoinverse  Image  Restoration 
William  K.  Pratt  and  Ikram  Abdou 

Andrews  and  Patterson  <1-3>  hare  recently  introduced  the  singular 
value  decomposition  (SVD)  of  matrices  as  a means  of  performing 
pseudoinverse  image  restoration  for  blurred  images.  Also,  Pratt  and 
Davarian  <4>  have  developed  a fast  computational  algorithm  for 
pseudoinverse  image  restoration.  This  report  proposes  an  application 
of  the  fast  pseudoinverse  algocitha  to  SVD  image  restoration. 

Pseudoinverse  Iaage  Restoration:  An  imaging  system  resulting  in 
the  observation  of  a spatially  blurred  iaage  in  the  presence  of 
additive  noise  is  often  modelled  by  a set  of  linear  equations.  The 
vector  space  representation  of  these  equations  is  given  by 

* = £ L * " (3.3-1) 

where  is  an  II  i 1 vector  of  pixels  obtained  by  sampling  and  column 
scanning  of  the  observed  iaage,  ^ is  an  I x 1 vector  of  points  of  an 
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ideal  iaage  field,  _B  is  the  H x I blue  aatrix  containing  eleaents 
which  are  saaples  of  tna  blur  iapolse  responsa  function,  and  jn  is  an 
R x 1 vector  representing  additive  observation  noise.  It  vill  be 

assuaed  that  the  staple  spacing  on  ^ and  is  the  sane  and  that  the 
iBpulse  response  is  space  invariant.  Purtheraore,  for  siaplicity  in 
explanation,  the  presentation  is  liaited  to  a one  diaensional  blur. 

Under  these  conditions  the  blur  aatrix  assaaes  the  fora 

t(L)  ....  h(ip  0 . . . . 0 1 

0 N h(L)  . . . h(l)“  N0.  . . 0 I 


(3.3-2) 


1^0  ...  . h(L) MlJ 

where  is  an  L x 1 vector  of  points  of  the  iapulse  response.  It 

should  be  noted  that  this  spates  is  underdeterained  since  N = H»L-1. 


The  pseudoinverse  estiaate  of  JL  is  given  by 


f * B | 


(3.3-3) 


where  _B~  denotes  the  generalized  inverse  of  The  generalized 

inverse  provides  a ainiaua  least  square  error,  ainiaua  non  restoration 
of  the  ideal  iaage.  If  is  a aatrix  of  rank  N the  generalized 
inverse  assuaes  the  fora 


B'  = BT(B  BV 


(3.3-4) 


ilgorithas  requiring  on  the  order  of  N operations  have  been  developed 


for  coaputation  of  the  generalized  inverse. 
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Tbe  fast  pseudoi averse  algoritha  <4>  is  based  upon  the 
developseat  of  an  adjoint  sodel 


Se  * £iE+.?E 


(3.3-5.) 


where  the  extended  vectors  ^E  and  _fE  are  defiaad  in  correspondence 
with 


M 


J-M 


J-K 


-T 


(3.  3- 5b) 


The  vector  *-s  coapjsai  of  tha  center  K pixels  of  the  ideal  inage 

vector  i_  where  K * M>2(L~1),  and  £ it  a circulant  Matrix  of  the  fors 

Ml)  0 . . 0 £<L)  . . . . h(2T 

M2)  hfl)'  N v 


h{L)  h(L-l) 

0'vh(L) 

• .V 

N 

• • N 


V 


N h(l*) 
0 


(3.3-6) 


> 0 . . Ovh(L)  ....  h(l) 

The  vector  *-s  identical  to  the  isage  observation  over  its  R center 
eleeents  where  R = N-2(L-1).  The  outer  eleaents  of  ^ can  be 
approxiaated  by  <b> 

q «q  = W g (3.3-7) 


where  Jj.  is  a windowing  aatrix.  Cosbining  egs.  (5)  and  (7),  an  estiaate 
of  can  be  obtained  froa 


iE*  £ 3e 


-1  ~ 


(3. 3-8a) 
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in  correspondence  vita 


K f. 


(3.  3- 8b) 


where X is  the  inverse  of  the  circulant  blar  latrix. 

The  restoration  operation  of  eg.  (8)  can  be  perforaed  indirectly 
by  Fourier  doaain  processing.  Letting  denote  the  Fourier  transfora 


aatrix,  eg.  (8)  can  be  rewritten  as 


where  both 


iE* 


C = A CA' 


(3.3-9) 


(3.  3- 10a) 


and  its  inverse 


or1  = A C~lA~l 


(3.  3- 10b) 


are  found  to  be  diagonal  aatrices  since  C_  and  SL  are  circulant, 
teres  of  C can  be  obtained  fron  <4>  * 


£ = diag[h  (J)] 


where 


L 


— E 


(3.3-12) 


J-L 


With  the  fast  pssudoinverse  algorithm,  ths  brute  force  Matrix 
operations  of  eq.  (9i  can  be  replaced  by  a pair  of  one-diaensional 
Fourier  transforas  and  J scalar  aultiply-aii  operations.  The 
computational  saving  is  usually  at  least  a factor  of  1000:11 


STD  laage  Bestoration:  The  blur  aatrix  jj  of  eg.  (1)  can  be 

decomposed  in  the  product  fora 


B = U VT 


(3.3-13) 


by  a singular  value  aatrix  decomposition  where  JJ  and  X are  unitary 
aatrices  composed  of  the  eigenvectors  of  BBT  and  J3  , respectively, 

and  X rs  a diagonal  aatrix  whose  diagonal  terms  are  the  eigenvalues  of 

T T 

JB  or  B.  B.  Since  X and  X are  unitary  aatrices  the  generalized 
inverse  of  JB.  can  be  wcitten  directly  as 


- _i  t 
B = V A 1 U 


(3.3-14) 


and  the  STD  pseudoinverse  estimate  assumes  the  fora 

£=  B“  g = VA-iu_T4 


(3.3-15) 


The  operations  of  eg.  (15)  may  be  viewed  as  a linear  transformation 

T - - 

(0  jj)  followed  by  a scalar  weighting  (multiplication  by  A 2)  , followed 
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by  another  linear  transforaation  (aultiplication  by  V)  . The 

transforaations  are  iato  the  eigen~space  of  the  blur  aatrix  _B. 

2 

Estimation  by  ey. (15)  requires  N aultiply  and  add  operations. 

As  a consequence  of  the  orthogonality  of  JJ,  and  it  is  possible 
to  express  the  blur  aatrix  in  the  series  fora 


B = ^ C X(i) 


(3.3-16) 


where  the  X (i)  are  diagonal  elements  of_h_,  aad  are  the  i th  rows 
of  if  and  ±,  respectively,  and  R is  the  rant  of  the  aatrix  JL.  The 

D 

generalized  inverse  of  J.  can  also  be  written  in  the  series  fora 


b"  = £ 

i=  1 


(3.3-17) 


Hence,  the  S7D  pseudoin  verse  estiaate  can  be  expressed  as 


*b 

A 'X.  ^ JL  rp 

f = / . CX(i)]"2  v,uT  g 


(3.3-18) 


Equation  (18)  can  ha  aanipulated  to  obtain  a sequential  SVD 
pseudoinverse  estiaate.  The  k th  estiaate  is  equal  to 


4 ■ 4-i  + [»»>]'*  <iTk  *4 


(3.  3-19) 


for  1 <L  k <_T  where  T represents  the  nuaber  of  teras  eaployed  in  the 
series  expansion.  3na  of  the  principal  advantages  of  the  sequential 
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formulation  is  that.  pcoolems  of  ill-conditioninj  generally  occur  only 
foe  the  higher  order  singular  values.  Thus,  it  is  possible  to 
terminate  the  expansion  before  nuaerical  probleis  occur.  The  aajor 
difficulty  associatai  with  the  sequential  STD  algoritha  is  the  large 
amount  of  computation  involved  in  generation  of  the  orthogonal  vectors 
and  _v  and  the  number  of  operations  required  to  implement  egs.  (14)  or 
(17)  . 


Past  STD  ilgoritam:  The  STD  pseudoinversion  technique  can  be 
applied  iirectly  to  tia  adjoint  model  of  eg. (5)  by  the  STD  expansion 


C = X £ y*t 

(3.  3-20) 

where  X and  X are  unitary  matrices  defined  by 

T *T 

X(CC_  ) X *A 

(3.  3-21a) 

T *T 

y(C  £)Y  * A 

(3.  3-21b) 

T 

Since  jC  is  circulant,  C,  _C  is  also  circulaot.  Therefore,  _X  and  JT  must 
be  equivalent  to  the  Fourier  transform  matrix  X (or  if)  since  the 
Pourier  matrix  perforas  a diagonalization  of  a circulant  matrix.  For 
purposes  of  standardization  let  X*  L*  1^*  As  a consequence,  the 
eigenvectors  x-  = , which  are  rows  of  X I»  ace  actually  the 

complex  exponential  basis  functions  of  a Fourier  transform.  That  is, 

x£(j)  “ (3.3-22) 


It  is  easy  to  show  that 
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A = CC*1  (3. 

where  C * A.  £ A.  * is  coaputed  frai  eqs.  (11)  and  (12)  . 

Then  in  correspondence  with  eg.  (15)  the  SVD  pseudoinverse 
estimate  of  the  extended  ideal  inage  vector  becones 

A . i ♦T  ** 

iE  = I £ * * aE  <»• 

or 

jE  = A'V*  A qE  (3. 

Equation  (24b)  should  oa  recognized  as  exactly  equivalent  to  the 
Fourier  donain  pseudoin versa  procedura  defined  in  eg. (9) . Hence,  it 
lay  be  concluded  that  the  STD  eigen  space  for  a circulant  aatrix  is 
the  Pouriar  donain.  Conputation  of  the  fast  STD  estinate  by  eg.  (24b) 
requires  J log2J  operations  each  for  the  forward  and  inverse  Fourier 
transforns  using  a fast  Fourier  transforn  (FPT)  algorithn  plus  j 
scalar  nultipliers  for  a total  of  J(1  ♦ 21og2J)  operations. 

A series  forn  of  tne  fast  STD  pseudoinverse  estinate  can  be 
written  directly  in  correspondence  with  eg.  (18)  as 

R 

iE=f>i,r±»*x[qE 

i=  1 

where  Hc  is  the  rank  of  £.  Sinilarly  in  correspondence  with  eg.  (18) 
the  sequential  pseudoinverse  estinate  assuses  the  fora 

\ * _fE  + c 8(k> r*  2eL< 


3-23) 


3-24a) 


3- 24b) 


3-25) 


(3.3-26) 
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M 


II 


foe  1 <_  k.  <_  T where  r c ^presents  the  nuaber  of  terns  eaployed  in  the 

series  expansion.  Jsiny  an  PPT  to  pre-coapute  and  store  the  product 
T~ 

[XkiEJ  results  in  a totil  of  TJ(1*log2J)  operations  required  for  the 
coaputation.  Greater  conputational  efficiency  can  be  realized  if  the 
series  is  sequentially  bisected  until  the  point  of  ill-conditioning  is 
reached.  This  strategy  requires  about  J log2 J[1*log2J ] operations. 


Past  Modified  SfP  Algoritha:  The  series  and  sequential  S¥D 

pseudoinverse  estiaates  of  eqs.  (24)  and  (25)  can  be  expressed  in 
equivalent  vector  space  fora  as 


where 


A4‘) 


*SE 


[5(1)1* 

[&(2  )]“* 


0 


(3.3-27) 


’ [6(T)T2 

0 


0 


(3.  3-28) 


This  estiaate  inherently  provides  a truncation  of  the  high  frequency 

terns  of  the  Pourisr  donain  pseudoinverse  operator.  Cosplete 

truncation  to  avoid  ill-conditioning  is  often  unnecessary.  As  an 

alternative  to  truncation  the  diagonal  zero  elenents  could  be  replaced 

_A 

by  unity,  or  the  valne  of  (6(k)]  5 or  perhaps  by  soae  sequence  that 


declines  in  value  as  a function  of  frequency 


Experiaental  Results:  Figures  1 and  2 contain  exaspl.es  of  the 
sodifiei  fast  S70  pseudoinverse  isage  restoration  aethod.  A blurred 
iaage  has  been  foraei  b / coaputer  siaulation  of  horizontal  blurring 
with  an  iapulse  response  of  the  fora 


h(l)  = 


where  a is  the  blur  standard  deviation  and  L is  the  length  of  the 

D 

iapulse  response.  The  restoration  algoritha  involves  repeated 
restoration  with  eg. (26)  in  which  the  J-T  zero  terns  along  the 

-l 

diagonal  are  replaced  by  [6(T)]2. 

In  figure  1 for  blurring  with  = 2,  there  is  a sudden  change  in 
the  ill-conditioning  effect  between  restorations  obtained  with  T = 89 
and  T * 90  aodified  Fourier  coefficients.  The  onset  of 

ill-conditioning  is  rot  guite  so  pronounced  in  the  sat  of  restorations 
of  figure  2 in  which  = 3. 
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Examples  cf  sequential  SVD  pseudoinverse  image 
restoration  for  horizontal  Gaussian  blur  with 
c,  -2,  L=  1 5,  J=256. 


Figure  3.  3-1 


(a)  Blurred  observation  (b)  Restoration 

T=  58 


(c)  Restoration 
T = 60 


Figure  3. 3-2. 


Examples  of  sequential  SVD  pseudoinverse  image 
restoration  for  horizontal  Gaussian  blur  with 
CT,  - 3,  L=23,  J=256. 
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3.4  Nonstationary  Hiaaer  Filtering  of  Film-Grain  Noise 
Ficouz  Naderi  and  Alexander  A.  Sawchuk 

Taking  into  account  the  cheaical  and  optical  degrading  factors  as 
veil  as  the  granularity  noise  results  in  a subjectively  reasonable 
aatheaatical  aodel  foe  the  formation  and  recording  of  photographic 
images.  This  model,  represents  highly  nonlinear  observations,  making 
the  subsaquent  restoration  difficult,  When  images  are  scanned  uith 
very  small  apertures  on  a nicrodensitoneter,  the  noise  is  so  severe 
that  conventional  estiaators  do  not  perform  wall.  In  these  cases 
detection  schemes  suited  to  individual  degraded  pictures  are  much  more 
effective.  When  the  noise  is  not  so  severe,  an  adaptive  minimum  mean 
square  error  filter  can  be  applied.  This  filter  explicitly  includes 
the  nonlinear  image  formation  effects  and  does  not  require  the 
assumption  of  stationary  image  statistics. 

Model;  Basic  to  all  estimation  problems  are  a signal  s to  be 
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estiaated,  a noise  n which  has  Bade  the  estiaation  necessary  and  an 
observation  which  is  a function  of  the  signal  ini  tha  noise.  Denoting 
the  observation  by  y let 

y=  g(«,n)  (3.4-1) 

where  eg.  (1)  is  the  observation  aodel.  Tha  function  g in  the 

observation  aodel  aay  or  aay  not  be  linear.  If  nonlinear*  the 
nonlinearity  aay  be  in  taras  of  the  signal  only  or  it  aay  include  soae 
cross-aultiplication  tecas  involving  the  signal  and  the  noise. 

The  statistics  of  the  noise  aay  depend  on  soae  paraaeter  of  the 
signal,  thus  aaking  it  signal-dependent  noise.  The  coaplexity  of 
different  types  of  astiaator3  designed  to  estiaate  _s.  froa  the 
observation  *s  * acy  auch  dependent  on  the  coaplexity  of  the 
observation  aodel.  To  illustrate  this  point,  consider  the  ainiaaa 
aean  square  error  (BHSB)  estiaate  of  Denoting  this  estiaate  by 
it  is  well  known  that 

a = E(aJjr)  = J » P^ljr)  d£  (3.4-2) 

where  P(sl^j  is  the  a posteriori  probability  density  function  of  £. 
Clearly  evaluation  of  the  NNSE  estiaate  of  requires  evaluation  of 
this  probability  density,  which  aay  prove  to  be  quite  difficult  if  the 
tunction  g in  eg. (1)  is  highly  nonlinear  in  nature. 

Thus,  before  attaapting  any  restoration  of  iaages  corrupted  by 
tila-grain  noise,  it  is  iaperative  to  establish  an  analytical  aodel 
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which  reasonably  reflects  the  formation  and  recording  of  iaages  and 
yet  is  eatheaatically  tractable. 


In  developing  a eathenatical  aodel  for  any  real  world  physical 
process,  one  faces  taa  tradeoff  between  the  utility  of  the  sinplified 
aodels  versus  the  accuracy  of  the  aore  coupler  ones.  In  the  case  of 
iaages,  the  degradation  in  the  recorded  inage  is  not  aerely  due  to 
granularity  noise,  but  is  also  caused  by  coupler  cheaical  and  optical 
effects. 

The  aodel  chosen  for  analysis,  shown  in  figure  1,  is  basically  an 
augaented  version  of  Kelly's  aodel  <1>.  Tha  aodel  of  figure  1 has 
been  subjectively  evaluated  by  coaparing  a real  world  iaage  with  a 
siailar  target  siauiated  according  to  this  aodel.  The  results  are 

shown  in  figure  2.  Figure  2a  shows  a Kodak  Pan-X  fila  digitized  using 

2 

a 2 (ua)  aperture.  Figures  2b  and  2c  show  siauiated  targets  of  alaost 
the  saae  size,  shape  and  contrast.  Figure  2c  was  siauiated  according 
to  the  the  aodel  in  figure  1,  while  in  figure  2b  only  the  effects  of 
granularity  were  considered. 

Restoration;  In  scanning  photographic  filas,  it  is  sonetiaes 

2 

accessary  to  use  very  snail  apertures  (on  the  order  of  4 to  25  (ua)  ) 
on  the  aicrodensi toaster  in  order  to  detect  fine  details  in  the  iaage. 
For  apertures  of  tais  size  the  fila-grain  noise  is  so  severe  that  in 
regions  of  low  contrast,  conventional  restoration  techniques  enploying 
linear  or  nonlinear  astiaators  will  have  little  effect. 

In  such  cases  algorithas  suited  to  a specific  degraded  photograph 


Figure  3.4-1.  Suggested  model  for  the  image  formation  and  recording  system 


I 


(a) 


(b) 


(c) 


Figure  3.4-2.  Subjective  evaluation  of  imaging  model  of  figure  1. 


\ 
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cau  be  developed  to  yield  very  good  results  <3-4>.  Oae  such  scheae  is 
the  noise-cheating  algorithm  devised  by  Zweig,  Barrett,  and  Hu  <2>. 
It  has  been  shown  that  this  algorithm  is  a sub-optimal 
maximum-likelihood  detection  scheme,  but  is  useful  as  the  basis  of  a 
lore  flexible  sub-optimal  Bayesian  detection  algorithm  <4,7-8>. 
Although  the  development  of  this  detector  will  not  be  discussed  here 
iu  detail,  its  effectiveness  is  demonstrated  in  reference  <7>.  If 
this  detection  scheae  i3  to  be  successful,  the  parameters  used  in 
designing  the  detector  need  to  be  accurately  calculated  based  on  each 
individual  picture  to  be  processed.  This  is  in  contrast  with  some 
purely  statistical  estimators  such  as  Wiener  filtering  in  which  the 
estimator  parameters  are  calculated  in  terms  of  the  ensemble 
statistics  of  a class  of  images  so  that  all  pictures  belonging  to  that 
class  can  be  processed  by  the  same  estimator.  The  tradeoff  is  that 
these  estimators  have  little  effect  when  the  noise  is  severe. 

Another  disadvantage  of  Wiener  filters  as  applied  to  images  is 
the  assumption  of  stationarity . In  general,  most  images  of  interest 
are  not  statistically  stationary.  Hunt  <5>  points  out  that  in  most 
cases  images  have  a stationary  second  moment  about  a non-stationary 
mean. 


A linear  NHSE  estiiator  based  on  the  model  of  figure  1 in  which 
the  image  is  not  required  to  be  stationary  will  now  be  derived. 
Figure  3 shows  the  discrete  version  of  the  model  in  figure  1 where 
is  an  H x > blur  matrix  which  accounts  for  the  combined  effects  of  bj 
and  b2.  Similarly,  J2  ie  * * * i blur  matrix  describing  blurs  b3  and 
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1 


b4  of  figure  1.  Operation  in  the  linear  region  of  the  H and  D curve 
is  assuaei.  Thus  tha  f auction  g in  figure  1 is  approximated  by  a 
linear  function  whose  parameters  area  and  J,.  Hatri*Xis  structured 
so  that  it  reflects  t a a nonlinearity  of  the  obsarvation  model  and  is 
of  the  form 


F = k 


(3.4-3) 


where  k is  a constant  ralated  to  the  grain  size  and  the  scanning 
aperture,  exponent  p is  a constant  which  different  experimentors  have 
reported  to  be  anywhere  in  the  range  0.2  - 0.5,  and  de1,...,de^  are 
elements  of  the  non-stationary  mean  vector  D of  figure  3.  Finally, 
vector  N is  taken  to  be  a zero-mean,  unit- variance  multivariate  normal 
vector  which  is  statistically  white  and  independent  of  the  signal. 


As  it  is  well  known,  the  linear  BHSK  estiaator  has  the  form 

* * (CID  > <CD  D >'* 
o o o 

regardless  of  the  underlying  observation  aodel,  where  and  £D  D are 

o o o 

the  cross-covariance  of  the  signal  and  observation,  and  the 
auto-covariance  matrix  of  the  observation,  respectively.  Evaluating 
the  above  two  matrices  from  figure  3,  it  can  be  shown  that 

5 - +JmBT  + B)  <a*H’  + C^.f1  H^1  (3.4-5) 


L.J 


r he  form  of  eg. (5)  is  greatly  simplified  by  setting  the  scaling  factor 
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a - 1 anl  0-0.  In  this  case 

2 * cuh1T<h1ci1i£  +sn.m,)-1i£i  <3-4-6' 

In  the  above  equation  JJq  is  the  auto-covariance  of  the  signal  I which 
is  to  be  estimated  aad  CN,N, is  the  au to-covarianca  of  the  effective 
noise  tere  which  is  a diagonal  eatrix  whose  diagonal  is  related  to  the 


(2p)th  assent  of  the  signal.  It  has  the  fora 

Diagonal  (C^,)  = k2(Hj)  (I)2p  (3.4-7) 

where 

(I.)2p  » E[ljP,  1*P 4p]  <3.4-8) 

and  where  ij,i2#...»iN  are  the  N eleaents  of  the  signal  vector  I. 

Mote  that  if  p * 0.5  then 

I2p  = I = E(I)  (3.4-9) 

ui 

where  _Jmis  the  non-3tationary  eean  of  the  signal  vector  _I.  In  this 


case  the  filter  JL  is  coapletely  defined  in  the  terns  of  the  first  two 
nonents  of  the  signal,  the  blur  natrices  Jflj  and  Ji2,  sad  the  scanning 
constant  k.  If  p is  tiken  to  be  a sore  reasonable  value  of  less  than 
0.5  then  the  (2p)th  aoaants  of  the  signal  are  required  to  construct 
the  filter.  It  should  be  noted  that  since  no  restriction  on 
atationarity  is  placed  on  the  signal,  then  vector  I ^ the  aean  of  the 
signal,  is  not  a constant  vector. 

hpplication:  This  filter  has  been  applied  to  one  dinensional 
vectors  of  length  123  x 1 as  shown  in  figure  4.  Figure  4a  is  the 
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original  which  was  seLected  fcoa  one  line  of  a digitized  iiage.  This 
represents  signal  figure  3,  which  was  then  blurred  by  a Gaussian 

blur  aatrix  with  a width  of  5 eleaents.  The  result  is  signal  JE  of 
figure  3 shown  in  figure  4b.  Since  a = 1 and  B.  - 0,  signals  E.  and  J2.e 
in  figure  3 are  identical.  Next,  signal-dependent  Gaussian  noise  is 
added  to  signal  0 to  obtain  signal  0 shown  in  figure  4c.  Finally  ]) 
is  blurred  with  blur  aatrix  iL2  with  a width  of  3 eleaents  to  produce 
the  observed  signal jfc  shown  in  figure  4d.  This  simulated  observation 
is  filtered  with  the  linear  MUSE  filter  discussed  above  to  produce  the 
astiaate  t_  shown  in  the  final  picture  of  figure  4.  The  saaple  aean 
sguare  error  (NSE)  was  reduced  froa  0.0433  in  the  observed  signal  to 
0.01798  in  the  estiaated  signal,  an  iaproveaent  by  a factor  of  2.4. 

It  is  interesting  to  coapare  the  perforaanca  of  this  filter  with 
a siailar  filter  waich  assuaas  the  signal  to  be  stationary  in  the 
aean.  Figure  5a  shows  a step  function  with  the  two  levels  set  at  0.4 
and  1.8.  Taking  this  as  the  original  signal  and  setting  blur  aatrices 
il1  and  ji2  to  the  identity  aatrix,  signal-dependant  noise  was  added  to 
the  step  function  to  obtain  the  signal  shown  in  figure  5b.  This 
signal  was  then  filtered  with  the  filter  discussed  here  as  well  as  a 
filter  which  assuaed  stationary  aean.  The  results  are  shown  in 
figures  5c  and  5d  respectively.  Saaple  NSE  showed  a 10%  iaproveaent 
when  no  restriction  on  stationarity  is  iaposed  on  the  signal. 

Application  of  this  filter  to  iaages  is  a straight-forward 
extension  of  the  one-diaensional  case.  The  inage  is  first 
lexicographically  arranged  in  a coluan  vector  to  obtain  a 


natrices  of  this  size,  one  of  two  options  are  usually  taken.  Both 
options  result  in  sub-optiaal  filtering.  The  first  option  is  to 
assuse  the  filter  is  separable  and  apply  it  to  rows  and  columns  of  the 
degraded  inage  sequentially.  The  second  option  is  to  break  the  isage 
into  snail  sub-blocks  (say  16  x 16  elenents)  and  then  process  each 
block  individually. 

A few  coaputatioaal  aspects  for  the  second  option  can  be 
considered  at  this  point.  Since  the  nean  of  the  inage  is  assuned 
non-stationary,  the  nean  of  each  individual  16  x 16  sub-block  differ 
fron  one  another.  Bxanining  eqs.  (6)  , (7)  and  (9)  it  is  clear  that  the 
natrix  CN,N, will  differ  for  each  16  x 16  sub-block  and  therefore  the 
filter  V of  eg. (6)  nust  be  calculated  for  each  block  separately. 

The  conputation  :ai  be  reduced  in  a nunber  of  ways  by  naking  sone 
sinple  assunptions.  3na  way  is  to  use  an  adaptive  schene  in  which  the 
filter  is  updated  froa  one  sub-block  to  another  only  if  a significant 
difference  exists  between  the  neans  of  the  two  sub-blocks.  A second 
nethod  i3  to  assuae  the  overall  nean  of  the  inage  is  globally 
non-stationary,  but  i3  locally  stationary  within  each  elenental 
16  x 16  block.  This  aains  that  within  each  block  the  nean  can  be 
taken  to  be  a constant.  Equations  (7)  and  (9)  indicate  that  if  this 
is  taken  to  be  the  c*3e,  then  natrix  £N,N,of  eg.  (6)  can  be  written  in 


the  forn 
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~N'N'  ‘ YU) 


(3.4-10) 


where  1_  is  the  ideutity  aatriz  and  y is  a constant.  In  this  case  the 
a uly  change  in  filter  JL  of  eg. (6)  fro*  one  block  to  another  is  caused 
. only  by  the  change  in  Y . Inspecting  eq.  (6)  it  is  noted  that  ,*ost  of 
the  calculation  in  foraing  filter  ji,  is  due  to  the  inversion 

- 2 + — N’N|)"1  (3.4-11) 

The  other  inversion  to  fora  J.2  needs  to  be  calculated  only  once  since 
ii2  does  not  change  froa  one  block  to  another. 

If  the  assuaption  of  local  stationarity  is  Bade  such  that  eq. (10) 

holds,  than  a siapla  way  exists  to  perfora  the  inversion  of  eq.  (11). 

T 

This  is  done  by  using  a siailarity  transf oraation  on  aatrix  (fliCjjfii) 
to  reduce  it  to  a diagonal  fora.  Since  aatrix  is  of  the  fora 

given  in  eq. (10)  the  siailarity  transforaation  would  leave  it  diagonal 
and  eg.  (11)  is  reducel  to  the  inversion  of  a diagonal  aatrix  at  each 
block.  This  filter  was  tested  on  a siaulated  iaage  and  the  results 
are  shown  in  figure  6. 

r 1 

Figure  6 represents  one  of  the  first  atteapts  in  applying  the 
discussed  MSB  filter  to  two  diaensional  signals.  In  obtaining  the 
noisy  iaage  of  figure  6 the  two  aatrices  JLj  and  JL2  were  set  to  the 
identity  aatrix.  is  dictated  by  the  non-stationary  Bean  of  the 
signal,  three  different  HHSE  filters  were  obtaihed  to  operate  on  the 
rows  of  the  noisy  iaage.  The  estiaated  iaage  is  shown  in  figure  6c. 

| I 


I 

I 


(a)  Ideal  image 


(b)  Noisy  image 


Figure  3.  4 


(c)  Estimated  image 


6.  Application  of  MMSE  filter  to  two  dimensional  signal. 
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More  results  in  application  of  this  filter  to  images  are 
forthcoming.  It  is  expected  that  if  filtering  is  done  on  sub-blocks 
rather  than  rows  of  tie  degraded  iaage  to  take  advantage  of  iaage 
correlation  in  bota  horizontal  and  vertical  direction  better  results 
can  be  obtained. 

Conclusions:  It  aas  been  deaonstrated  that  when  dealing  with  low 
contrast  inages  and  severe  noise,  sone  picture  dependent  detection 
scheaes  perform  better  than  filters  designed  solely  on  ensenble 
statistics.  It  has  also  been  shown  that  there  is  an  advantage  in 
designing  HHSE  filters  with  no  stationarity  of  the  aean  inposed  on  the 
iaage. 

References 

1.  O.H.  Kelly,  "Systeas  Analysis  of  Photographic  Processing  I. A. 
Three-Stage  Model,"  Journal  Optical  Society  of  America,  Vol.  50, 
1960,  p.  259. 

2.  H.J.  Zweig,  E.3.  Barrett  and  P.C.  Hu,  "Noise-Cheating  Iaage 
Enhancement,"  Journal  Optical  Society  of  Aaarica,  Vol.  65,  1975,  p. 
1347. 

3.  H.J.  Zweig,  E.B.  Barrett  and  P.C.  Hu,  "Coaparison  of 
Noise-Cheating  and  tfiaoer  Filtering,"  Journal  Optical  Society  of 
Aaerica,  Vol.  65,  1975,  p.  1203A. 


4.  F.  Naderi  and  A. A.  Sawchuk,  "Nonlinear  Digital  Processing  of 


1 


Page  162 


Images  Corrupted  by  Pile-Grain  Noise,"  Journal  Optical  Society  of 
America,  Vol.  65,  1975,  p.  12021. 


5.  B.R.  Hunt,  "Digital  Image  Processing,"  Proceedings  IEEE,  Vol. 
63,  1975,  p.  693. 

6.  H.C.  Andrews,  Coiputer  Techniques  in  Image  Processing,  Chapters  5 
and  6,  Academic  Press,  daw  York,  1970. 

7.  F.  Nader i and  A. A.  Sawchuk,  "Detection  and  Estieation  of  Images 
Degraded  by  Pila-Srain  Noise,"  OSC  Iaage  Processing  Institute, 
Semiannual  Technical  Report  620,  September  30,  1975,  pp.  92-100. 

8.  F.  Naderi  and  A. A.  Sawchuk,  "Nonlinear  Detection  and  Estimation 
of  Images  Degraded  by  Film-Grain  Noise,"  Proceedings  Optical  Society 
of  America  Topical  Heating  on  Image  Processing,  Asiloaar,  California, 
February  1975,  Journal  Optical  Society  of  America,  Vol.  66,  1976,  p. 


167. 


! 


3.5  Degrees  of  Freedom  in  Image  Formation 

Dennis  McCaughey  and  Harry  C.  Andrews 

The  concept  of  degrees  of  freedom  (DOF)  has  arisen  in  the  imaging 
literature  in  an  attaipt  to  guantify  the  number  of  truly  independent 
samples  of  data  one  gathers  with  photographic  or  othar  image  sensing 
devices  <1,2>.  As  sensor  technology  grows,  the  guantity  of  data 


gathered  increases,  and  it  becomes  reasonable  to  ask  what  the  true 
increase  in  information  content  is  as  one  increases  image  samples. 
One  useful  model,  in  which  the  degrees  of  freedom  (DOF)  concept  fits 
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uicely,  is  the  couti  mous-discrete  representation  of  a two  variable 
Fredhola  integral  <3> 

8i  =/ f f(§,Ti)hi(?,ti)d?dn  (3.5-1) 

here  the  object  f(?,ru  is  continuous,  and  therefore,  represents  a 
non-countably  infinite  nuaber  of  OOP.  The  inage  is  described  by  a 
finite  nuaber  of  saaples  H,  and  at  best  represents  N DOF.  However, 
the  inaging  integral  equation  causes  reduction  of  this  nuaber  due  to 
the  point  spread  function  (PSP)  blur  provided  by  h.  (?,r|)  . Here  the 
PSF  is  described  by  an  H-vector  whose  eleaents  are  continuous 
functions  of  the  object  coordinate  systea  (?,ri).  intuition  night 
serve  here  in  the  concept  that  the  greater  the  blur,  the  fewer  the 
OOP.  Thus,  if  h.  ( § ,t)j  is  a "narrow"  function  the  DOP  would  be  greater 
than  if  h^?,^)  were  a "broad"  function.  Since  h.  (?,t|)  can  represent 
a space  variant  point  spread  function  (SVPSP) , intuition  also  suggests 
that  in  regions  in  which  the  object  is  in  "better  focus"  (i.e. 

narrower  PSP)  greatec  DDP  will  be  obtained  than  in  regions  of  "poorer 

focus"  (i.e.  broadar  or  greater  blur  PSP).  Pigure  1 presents  the 

conceptualization  in  pictorial  fore  where  the  shaded  region  represents 

* 

the  aeount  of  overlap  of  neighboring  PSP's. 

To  place  these  iiaas  on  a firaer  aatheaatical  footing,  it  is 
suggested  that  the  Sraaian  aatrix  foraed  froa  the  PSP  R-vector  could 
provide  a guantitativa  aeasure  of  the  DOP  available  in  an  iaaging 

systea.  The  entrias  in  this  aatrix  represent  the  correlation  or 
*This  figure  has  been  provided  by  Dr.  B.S.  Hou  <Q>  whose  research 

aided  the  authors  in  developing  soae  of  these  concepts. 
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overlap  of  each  PSP  Kith  its  neighbors,  and  the  rank  and/or  eigenvalue 
nap  of  such  a aatrix  could  be  used  in  the  definition  of  the  DOP 
<1,2,5>.  Twoaey  has  aguated  the  degrees  of  freedon  of  an  inaging 
systea  with  the  nuaber  of  effectively  non-zero  eigenvalues  of  the 
Sraaian  of  the  iaaging  systea  <2>.  Tvoaey's  techniques  were  developed 
for  one-diaensional  systeas,  but  are  applicable  to  tvo-diaensional 
systeas  and  will  be  suitably  extended  and  applied  to  projection 
iaaging  in  what  follows. 


Continuous-Discrete  Model  and  the  Sraaian:  In  applying 
two-diaensional  linear  systeas  nethods  to  iaaging  systeas  the 
assuaption  is  nade  teat  an  iaage  g is  related  to  the  original  object  f 
by  a superposition  integral 


R(x*y) 


=//* 


h(*,y;§,ri)f(§,Ti)d§  dr\ 


(3.  5-2) 


where  h(z,y;5,Ti)  represents  the  weighting  function  of  the  iaaging 

systea  and  B is  the  region  of  integration  over  the  input  coordinate 

systea.  The  aia  of  any  reconstruction  process  is  to  recover  f as  best 

as  possible  in  soae  sense,  knowing  h and  g.  In  a practical  situation 

g is  known  only  for  a finite  set  of  points  in  the  output  plane.  Let 

g denote  g (x,y)  saapled  in  the  iaage  coordinate  plane  (x,y)  and 
i 

siailarly  for  h,  then  the  vector  fora  of  equation  (1),  the 
continuous-discrete  iaaging  aodel 


•■//*  ?#Tl)f(5,Tl)d§dTl  +S 


(3.  5-3) 


results  where  is  an  error  tera  and  all  vectors  are  N x 1 coluan 


vectors 
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In  obtaining  an  astiaate  f for  f in  eg. (3) , consideration  aust  be 
taken  of  the  fact  that  the  PSP  kernel  vector  is  coaprised  of  kernel 
functions  that  are  not  necessarily  independent.  Neglecting  the  error 
tern  for  the  aoaent,  tais  dependency  of  kernel  functions  iaplies  that 
sons  of  the  output  saaples  can  be  predicted  by  linear  coabinations  of 
the  others,  are  thus  superfluous,  and  serve  to  reduce  the  DOF  of  the 
inaging  systea.  The  arror  tera  aggravates  this  situation  since,  if  an 
output  elaaent  can  ba  predicted  by  a linear  coabination  of  the  others 
to  within  an  accuracy  better  than  the  aeasureaent  error,  that 
aeasureaent  adds  no  new  inforaation  to  what  is  already  known.  Twoney 
applied  this  reasoning  to  systens  involving  ona-diaeasional  integral 
equations  with  significant  results  <2>. 


To  obtain  a better  feeling  for  how  the  DDF  of  the  systea  aight  be 
reduced  due  to  this  linear  dependence  anongst  PSP  kernels,  an 
arbitrary  unit  length  factor  is  defined  such  that  its  dot  product  with 
the  iaaging  equation  yields 


<a,g>  =<  a,  //  M?,n)f(?,T1)d?dn>  +<a,d> 


(3. 5-4) 


R 

If  the  first  tern  on  tha  right  is  each  snaller  than  the  second,  then 
its  contrioution  can  ba  considered  niniaal.  Taking  this  into  account 
along  with  the  fact  that  Hall  » 1,  it  is  possible  to  reduce  the  degree 
of  independence  or  nuaber  of  DOP  by  one  for  every  independent  a such 
t hat 

h-/ (3 

R 

2 

where  j|<f||  can  be  considered  the  sensor  noise  which  contributes  to  the 
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definition  of  the  signal  to  noise  ratio  of  the  iaaging  systea.  The 
implications  of  eg.  (5 1 can  be  farther  investigated  by  noting  that  the 
vector  inner  product  on  the  left  can  be  related  to  the  continuous 
inner  product  of  f vith  h by 


*nK(«»  -n)d«  4v)  H*  =<f(S,Ti),<h(S,Ti),a»2 
R 

and  applying  the  Schwartz  iaeguality 

< I (§ . ti  ),  < h(?,  it),  a »2  ^ < fj 5 , n ),  , r\  )><  <h( S,  ri)#^  >,  < h(  5 , ti  ),  a » 


(3. 5-6) 


= IU(S,n)lf  a^ra 

Here  T is  known  as  the  Iranian  of  the  PSP  kernels  and  is  defined  as 


(3.  5-7) 


I =[//  V5'Tl)hf(?'Tl)d?dT'] 


(3.  5-8) 


which  is  Haraitian  having  real  eigenvalues  only.  Since  f(?#ri)  can  be 
considered  bounded,  |f(?#r|)|  £ E,  eg.  (5)  becoaes 


E2a*ra  ^ Hill2 


(3.  5-9) 


-v 

Expanding  £ into  its  sigenvector  decomposition  £ = JjAo.  and  letting 
1 - Jig,  another  unit  vector,  gives 


EZ1*A8S  ||£ II2 


(3.  5-10) 


Clearly  J3'A§_  is  ainiiized  by  the  allowing  £ to  be  the  eigenvector 
associated  with  tha  saallest  eigenvalue  of  £ ; which  leads  to  the 
conclusion,  that  for  svery  eigenvalue  of  £ 3uch  that 

e\  < II*  II2  ,J. 
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L 

I 


one  degree  of  freedoa  is  lost. 


Attention  can  now  be  directed  to  the  estinate  of  the 

A 

reconstruction  of  tha  object  f by  expanding  this  estinate  in  a set  of 
M orthogonal  functions  such  that 

N 

*(S.ti)=^  ri)  = <v(?,ri)fa  > (3.5-12) 

i=  l 


In  order  that  f be  a solution  up  to  the  degree  of  freedoa  of  r the 
orthogonal  set  will  be  defined  as 


2(S,n)  = 

Then 

(3.  5-13) 

//l«W  (S,Ti)d§dn  =y* y*  U^(5.ri)h*(?,Ti)Ud?dTi  = U*ru  = A 
R R 

(3.  5-14) 

Thus,  tha  set  is  ortaogonal.  Solving  for  the  iaaging  equation 
without  noise  yields 

for  f 

£ ~J“ n)«P*(§,T))a  d?dt) 

R 

(3.  5- 15a) 

g=f  f Ml,  T))h*(§  , T))U  ad§  dr] 

R 

(3.  5- 15b) 

= r u a = U AU*U  a = UAa 

(3.  5-15c) 

Consequently  the  coefficients  of  the  estinate  f froa  eg. (12)  becoae 
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a = A"1  U*  g 


Thus, 


f(§,Ti)=  t*r’xh(§,n) 


and  the  estimate  of  taa  object  is  shown  to  be  directly  related  to  the 
image  and  expansion  of  the  PSF  kernel  in  the  space  of  the  Granian 
(weighted  by  the  inverse  of  the  Granian  eigenvalues).  Clearly,  the 
larger  the  condition  number  of  £ , the  poorer  the  estinate  of  the 
object.  A large  condition  nuaber  on  r iaplies  low  eigenvalues  which 
iaplies  an  effectiva  linear  dependence  in  the  point  spread  function 
kernels  which  iaplies  overlapping  point  spread  functions  in  the 
iaaging  aodel. 

The  results  of  e^s.  (16)  and  (17)  indicate  an  ill-condit: oning 
problem  in  which  an  upper  bound  on  the  relative  error  ||  6 a||/l|a||  can  be 
found  to  be 


11**11 


11*11 

fig  II 


! Hi' 


Thus  the  condition  nuaber  of  T and  the  nois9  to  signal  ratio  in  the 
data  define  an  upper  sound  on  the  relative  error  in  the  reconstruction 
process. 

Heretofore  consideration  has  been  given  to  the  case  of  a 
non-singular  Graaiaa,  and  attention  aust  be  given  to  the  case  where 
A-1,  and  thusT-1,  do  not  exist.  Letting  C be  a non-singular  constraint 
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aatrix  this  problea  ran  be  postulated  as  a suitable  ainiaization  as 
follows  <13>: 


minimise:  a*  C a 


subject  to:  ||f  - U A a||2  =0 


(3. 

(3. 


The  criterion  to  be  ainiaized  becoaes 


a*  C a + Y ||g  - U A a||‘ 


(3. 


where  Y is  the  usjaL  Lagrange  aultiplier.  Perforaing 
ainiaization  results  in  a least  sguares  solution  of 


this 


a = (Y^c  + A2)" 1 A U*g 


(3. 


and 


f(5,t])  = » U*h(?,n) 


(3. 


the  above  result  should  be  coapared  with  egs. (15)  ani  (17)  to  see  the 
iaplications  of  thi3  Least  sguares  result.  Proper  conditions  on  C and 
y result  in  the  pseudoiaverse  reconstruction. 
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3.6  Texture  Image  Coding 
William  K.  Pratt 

H&ny  images  can  be  segmented  into  a relatively  few  number  of 
regions.  Some  regions,  called  busy  regions,  contain  objects 
possessing  fine  detail.  Other  regions  relatively  devoid  of  detail, 
can  often  be  described  as  being  of  similar  composition  or  texture, 
featured  regions  are  characterized  by  some  guasi-periodic  repetition 
>f  a basic  pattern  or  structure.  Examples  include  grass,  farm  fields, 

gravel,  tiles,  brick  walls,  and  textiles. 

\ 
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Coding  Concept:  Segmentation  of  an  image  into  busy  regions  and 
textured  regions  cia  be  utilized  as  the  basis  of  an  image  coding 
concept.  The  coding  process  is  as  follows: 


(a)  an  image  is  sagaented  into  two  region  classes:  textured  and 

busy; 

(b)  busy  regions  ara  identified,  and  pixels  within  the  region  are 
coded  by  some  high  quality  coding  method  such  as  predictive  or 
transform  coding; 

(c)  texture  is  characterized  according  to  some  measure  within 
each  textured  region; 

(d)  at  the  image  destination  busy  regions  are  reconstructed  from 
their  coded  representations; 


(e)  the  textured  regions  are  reconstructed  at  the  receiver  by 
inserting  artificially  generated  texture  within  each  texture 
segment. 


This  texture  coding  process  seeks  to  exploit  the  limitations  and 
adaptability  of  the  human  visual  process.  In  most  scenes  textured 
regions  simply  provide  the  spatial  context  for  fine  detail  busy 
regions.  The  visual  system  does  not  closely  examine  textured  regions; 
they  are  perceived  on  some  global  basis.  In  effect  the  fine  structure 
of  a textured  region  is  redundant  to  the  human  visual  systems.  This 
visual  property  has  bmea  exploited  by  artists  for  centuries.  In  aany 
paintings  textured  regions  are  represented  by  relatively  few  brush 
strokes  in  comparison  to  regions  of  fine  detail. 


Segmentation:  Tha  texture  coding  system  requires  segaentation  of 


an  iaage  into  busy  and  textured  regions  prior  to  coding  of  each 
region.  Iaage  segaantation  for  purposes  of  scene  analysis  la 
presently  an  active  field  of  research.  Techniques  under  study 
include: 

thresholding 
histograa  division 
clustering 
region  growing 
edge  linking 

Thresholding  is  a siapla  fora  of  segaentation  in  which  individual 
pixels  are  classified  according  to  their  aaplitudo  value.  The  process 
is  liaited  to  siaply  structured  iaages  for  which  the  textured 
background  possesses  an  average  brightness  level  significantly 
different  froa  the  busy  regions.  Histograa  division  is  a 
aulti-diaensional  fora  of  thresholding  based  upon  the  foraatiou  of 
histograis  of  iaage  attributes  such  as  luminance,  tristinulus  value, 
edge  density,  etc.  If  a histograa  is  aulti-aodal,  the  division  levels 
between  the  nodes  are  used  to  define  classification  cells  in  the 
aulti-diaension  iaage  attribute  space.  Clustering  techniques  of 
pattern  recognition  are  a generalization  of  histograa  division  in 
which  classification  caJLls  are  foraed  froa  the  joint  histograas  of  the 
iaage  attributes  ratnar  than  the  aarginal  histograa  of  each  attribute. 
The  histograa  and  clustering  approaches  often  perfora  guite  well  in 
segaentation,  but  taa  associated  processing  task  is  fornidable. 
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Begion  growing  is  a siaple  approach  to  segaentation  in  which 
neighboring  pixels  are  grouped  into  a region  if  they  possess  siailar 
attributes.  The  aethod  works  well  for  siaple  scenes,  but  aust  be 
augaented  with  seaantic  knowledge  for  reasonably  coupler  scenes.  Edge 
linking  is  a boundary  foraation  process  in  which  edge  points  are 
individually  detected  and  subsequently  linked  to  their  neighbors  if 
tney  possess  siailar  gradients  and  orientations.  The  aajor  drawback 
of  the  technigue  is  that  the  detected  boundaries  aay  not  be  closed. 
Further  work  is  needed  toward  the  developaent  of  segaentation  aethods 
which  can  be  utilized  in  the  texture  coding  systea. 

Texture  Synthesis:  There  are  two  basic  approaches  to  texture 
synthesis:  deteraini stic  and  statistical.  Bith  the  deterainistic 
method  a basic  patters  of  pixels  is  periodically  repeated  in  two 
diaensions  over  an  iaage  field.  The  pattern  aay  be  an  artificial 
arrangeaant  of  pixel  aaplitude  values  or  soae  saall  segaent  of  a 
natural  scene.  The  statistical  approach  involves  the  generation  of  a 
spatially  correlated  two  diaensional  randoa  process. 

Figure  1 contains  exaaples  of  artifically  generated  textures 
whose  average  luainance  level  is  200  over  a scale  of  0 to  255.  The 
first  exaapie  in  figura  la  consists  of  a spatially  uncorrelated  array 
generated  by  Gaussiauly  distributed  randoa  nuabers  with  standard 
deviations  of  32.  In  figure  1b  the  array  of  figure  la  has  been  low 
pass  filtered  by  convolution  with  an  iapulse  response  function 


(a)  Gaussian  density 
Y=200, o =16 

y 


(b)  Low  pass  filtering  of  (a) 


(c)  High  pass  filtering  of  (a) 
Pc  = Pr  = 0.99 


Figure  3.6-1.  Artificially  generated  textures 
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1 1 1 


(3.6-1) 


The  array  of  figure  Is  results  froa  high  pass  filtering  with  an 


iapulse  response  function 


P P 
C R 


"c(1+V> 


'c'r 


h=  -pRu+pc2)  u+*n2)(i+^2>  -vi+pc2) 


>lC*R. 


- 'C(1+  R ' 


P P 

CR 


(3.6-2) 


where  Pc  * 0.5  and  pR  * 0.9. 


Experimental  Besults:  The  basic  concept  of  texture  coding  has 


been  explored  for  tha  coding  of  the  tank  iaage  of  figure  2a.  The 


outline  of  the  tank,  shown  in  figure  2b,  has  bean  obtained  nanually  to 


provide  the  basis  for  the  segsentation  of  tha  tank  and  the  grass.  In 


figure  2c  the  tank  segment  has  been  combined  with  a constant  lusinance 


background  of  value  200.  The  reconstruction  of  figure  2d  utilizes  the 


low  pass  filtered  Gaussian  texture  array. 


The  results  presented  here  are  serely  a preliminary  example  of 


the  general  concept  of  texture  coding.  Effort  is  continuing  on 


several  fronts  to  completely  develop  the  technique.  Research  is  being 


performed  on  texture  segmentation,  measurement,  and  synthesis. 


3.7  Algorithms  and  Noi-ideal  Effects  in  Nonlinaar  Optical  Processing 


Stephen  R.  Dashiall  and  Alexander  A.  Sawchuk 


f preceding  page  blank- not  filmed 


Hi m 


(b)  Tank  outline 


(a)  Tank 


(d)  Reconstruction  with 
texture  background 


(c)  Reconstruction  with 
grey  background 


Figure  3.6-2.  Example  ot  texture  coding 
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The  use  of  halftoue  screen  preprocessing  to  realize  nonlinear 
operations  in  a coherent  optical  systen  is  well  established  <1-7>. 
Various  operations  such  as  logarithn,  exponential,  level  slicer,  notch 
filter,  quantization,  and  analog-to-digital  conversion  have  been 
presented,  (then  halftone  screens  are  used  to  realise  nonotonic  or 
non-nonotonic  nonlinear  point  transfer  functions,  the  screens  are 
usually  conposed  of  periodically  repeated  cells  which  are  thenselves 
nonotonic  in  density.  This  is  not  necessary:  an  iterative  conputer 
algorithi  for  generating  a non-aonotonic  cell  shape  to  yield  an 
arbitrary  transfer  function  in  the  first  orler  has  been  developed. 
There  is  no  unique  solution  to  the  non-nonotonic  design  problen.  The 
cell  generated  by  the  algorithn  is  only  one  of  nany  possibilities. 
Soee  operations,  specifically  the  notch  filter  and  quantizer  <2,4,6,7> 
are  realized  with  non-aonotonic  halftone  cells. 


Onitting  vavelengtn  and  geoaetrical  factors  for  clarity,  the  bar 
structure  of  period  a on  the  copy  filn  resulting  fron  using  a 
one-diaensional  halftone  screen  can  be  analyzed  by  representing  one 
period  (J  (x)  as  a sua  of  snail  shifted  rectangle  functions.  If  there 
are  snail  rectangle  functions,  then 

U(x)  = £ I(k, IJ  r.ct( -W**  a/2M  (3. 7-t) 

k=  1 ' / 

where  l can  be  arbitrarily  large,  * 0 if  the  specific 

sub-region  is  opaque,  and  I(k,In)  * 1 if  it  is  clear.  Equation  (1)  is 
convolved  with  a conn  function  <8>  to  produce  the  conplete  periodic 

H 4 

transai ttance  function  input  to  the  optical  processor  After  Fourier 


r 
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transforaing,  diffraction  order  selection,  anl  discarding  of  coaplez 
constants,  the  invarsa  transforn  is  forned  by  the  second  lens  in  the 
systea  and  the  aagnituda  squared  yields  the  output  intensity 


' 

' 

t! 


t 


l 

tout  = g(Iin,n)  = It  2 I(k* ^ exP  (-j2n kn/t)  | (3.7-2) 

k=  1 

whare  n represents  the  lif fraction  order  selected.  For  the  first 

order,  n = 1 . In  both  egs.  (1)  and  (2),  tha  tera  I(k,I  ) reflects 

in 

the  fact  the  the  sub-regions  depend  on  I through  the  halftone 
preprocessing. 

The  synthesis  problea  for  an  arbitrary  diffraction  order  can  be 
3tated:  choose  a function  I(k,l  ) 30  that  the  square  of  the  n th 

Fourier  coaponent  of  is  equal  to  the  desired  g(^n,n)  with  the 

restriction  that  if  I(k,In)  equals  one  for  any  I-n#  it  aust  be  equal  to 
one  for  all  larger  I.  . The  constraint  arises  because  no  sub-region 

1X1 

can  return  to  clear  after  being  exposed. 


Considering  only  the  first  order,  eq.  (2)  becoaes 

l 

Jout  = g(Iin#  = I I T.  [ (3.7-3) 

k=  1 


Ibis  can  be  regarded  as  a sua  of  l vectors  of  length  3 or  1/-L  with  the 
angular  orientation  of  the  h th  vector  given  by 


0(k)  = 2nk 
l 


(3.  7-4) 


I 


These  are  directly  equivalent  to  positions  ka/£  in  the  periodic  cell 
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A synthesis  algorithn  for  non-nonotonic  cells  to  achieve  arbitrary 

non-nonotonic  nonlinear  transfer  functions  can  be  foreulated.  The 

procedure  is  to  break  the  vectors  into  two  fanilies,  those  for  which 

1 < k < -t/2  and  thosa  Ear  which  (£/2)*1  < k < l , and  regard  the  first 

fanily  as  the  one  which  increases  the  output,  and  the  second  as  the 

one  which  decreases  tae  output.  To  begin,  add  in  vectors  starting 

with  k » 1 until  tha  sun  squared  equals  the  desired  IQut  for  the  1^. 

To  increase  I for  a larger  add  in  nore  vectors  one  at  a tine  and 

continue  until  the  sua  squared  equals  the  desirad  I , To  decrease  I 

out  out 

tor  a larger  Ln,  add  in  vectors  starting  with  k = l/2*^  or,  if  soae  of 
the  second  fanily  hava  already  been  used,  start  at  the  vector  adjacent 
to  the  one  last  used.  Add  these  in  until  the  sua  squared  has 
decreased  to  the  dasired  value.  Continue  this  process  until  the 

coapleta  nonlinear  transfer  function  has  been  napped  out.  One  of  two 
possibilities  will  occur:  (1)  all  of  the  vectors  in  the  first  fanily 
will  be  used  before  the  conplete  transfer  function  has  been  napped 
out;  or  (2) , the  function  will  be  conpletely  napped  and  there  will  be 

vectors  renaining  which  could  have  been  used.  If  the  first 

possibility  occurs,  increase  the  length  of  all  the  vectors  and  start 
over.  If  the  second  possibility  occurs,  decide  if  a large  enough 
fraction  of  the  available  vectors  (cell  area)  have  been  used,  and 
stop.  If  not,  decrease  the  vector  length  and  begin  again.  If  the 

tolerances  are  chosen  wisely,  this  procedure  will  arrive  at  a 
satisfactory  solution  fairly  rapidly.  Figure  la  shows  the  input 
picture,  and  figure  1b  shows  the  output  picture  for  a first  order 

intensity  notch  filtec  with  a non-nonotonic  cell  halftone  screen. 


Figure  3.7-1.  Nonlinearities  from  non-monotonic  halftone  cells. 
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Figure  1c  shows  the  output  of  a three  level  quantizer.  Figure  2 shows 
the  transfer  function  and  cell  shape  for  a first  order  five  level 
slicer. 

In  the  analyses  to  date  <1-7>  the  file  used  for  the  copy  step  in 
the  halftone  process  has  been  assuned  to  have  an  infinite  ganna  and 
saturation  density.  In  practice  neither  of  these  is  true,  and  the 
affects  of  low  gaiaa  or  saturation  density  affect  the  transfer 
functions  obtained  in  various  ways,  depending  on  the  particular 
halftone  screen.  A a analysis  of  the  effects  of  finite  gaiaa  and 
saturation  density  for  one-fila  and  two«fila  processes  will  be  given. 
A ene-filn  process  is  the  noraal  halftoning  process.  A two-filn 
process  is  the  saae  accept  that  the  halftone!  picture  is  contact 
printed  onto  another  piece  of  copy  fila  to  produce  a contrast  reversed 
halftone!  picture.  T»i3  is  useful  to  provide  a positive  slope  on 
aonotonic  transfer  functions  in  the  zero  order  and/or  increase 
effective  gaana  if  both  gaaaas  exceed  one. 

Errors  in  the  nonlinear  intensity  transfer  function  arise  froa 
the  edges  of  the  bars  on  the  copy  fila  being  varying  shades  of  grey 
with  a finite  width  rather  than  abrupt  juaps,  and  froa  the  finite 
transaission  of  supposedly  opaque  areas  on  the  fila.  A plot  of  the 
fila  density  across  one  bar  will  show  sloping  rather  than  abrupt 
sides,  and  a aaxiaua  density  equal  to  the  saturation  density  for  the 
fila.  If  this  bar  density  is  regarded  as  consisting  of  a stack  of  k 
trapezoids  each  of  height  AD,  the  output  intensity  resulting  fron 
using  an  arbitrary  n tb  order  coaponent  of  the  Fourier  transfora  of  a 
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processing  system  is 
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where  D ^ = halftone  call  density  corresponding  to  this  bar  width 


(3.7-5) 


D , = fils  saturation  density 
■at 

D'  = Dsator  Y Dj » whichever  is  saaller 
k * number  of  stacked  trapezoids  used 
n = diffraction  order  used 

f 1 = inverse  of  f where  f (c)  = D is  the  cell  profile  description 
f _D' 

DC  3 \1 0 for  a = 0 and  0 for  n 3 0. 


Realizing  that  Ijn  = log^j  , eg.  (5)  can  be  evaluated  for  various  Dj  and 

a picture  of  the  actual  I . for  various  D . and  y values  obtained. 
r out  mat  ' 

Equation  (5)  describes  the  effects  for  the  one-film  case.  For 
the  two-film  case 


where  AD  « D'/KY2 

D>at 2*  saturation  iansity  of  fila  no.  2 
Io*io 


n 
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Equation  (6)  is  used  to  calculate  a point  by  point  plot  of  I for 
various  i.q  values,  given  the  other  specified  parameters. 


Equations  (5)  and  (6)  have  such  a large  nuaber  of  variables  that 
a conprehensive  secies  of  graphs  showing  all  possible  behavior  nodes 
is  inpossible  in  the  space  available.  The  plots  shown  assune  high 
saturation  density  and  low  gannas.  Figures  3 to  6 show  ezanples  of 
one  and  two  filn  transfer  functions. 

It  is  possible  to  aininize  certain  degradations  caused  by  finite 
ganna  and  saturatioa  densities  by  preconpensation  of  the  halftone 
cell.  The  preconpensation  can  be  accomplished  for  portions  of  the 
transfer  function  waich  are  neither  flat  nor  an  abrupt  juap.  The 
technique  is  as  follows: 

(a)  Calculate  or  specify  the  ideal  transfer  function. 

(b)  Using  the  specified  filn  paraaeters,  calculate  point  by  point 
what  the  required  bar  width  is  to  obtain  the  desired  output  using 
eg.  (5)  or  (6)  as  appropriate,  with  the  restriction  that  the 
calculated  cell  nust  be  nonotonic,  i.s.,  f ( z)  * D is  a 
aonotonically  increasing  function  of  z. 

Mote  that  when  the  aaziaun  allowable  bar  width  is  reached,  further 
increases  in  I serve  sinply  to  increase  the  overall  filn  density. 

Figures  7 and  8 show  sone  results  fron  this  technique.  Mote  from 


! 


j 


figure  8 that  even  with  low  gaana  and  saturation  density,  a good  natch 
to  the  desired  transfer  function  is  possible. 
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3.8  Artificial  Stereo  froa  Iaage  Features 

Alexander  A.  Sawcbuk  and  Harry  C.  Andrews 

It  is  well  known  that  stereo  iaage  inforaation  is  an  absolute 
necessity  for  certain  iaage  processing  applications  such  as 
photointerpretation  and  cartography.  This  section  describes  soae 
preliainary  experiaants  in  artificially  generating  stereo  iaage  pairs 
froa  single  pictures.  In  these  artificial  stereo  iaages,  the  depth 
inforaation  can  be  obtained  froa  aany  different  types  of  iaage 
features,  including  pixel  brightness,  edge  inforaation,  texture,  or  a 
coabination  of  these.  One  intent  of  this  study  is  to  evaluate  the 
depth  illasion  created  by  artificial  stereo  as  an  aid  to  huaan 
perception.  An  iaportant  reference  for  the  study  of  textures  in 
stereo  perception  is  the  work  of  Julesz  <1>. 

A prograa  has  been  written  to  accept  any  height  inforaation  about 
a two-diaensional  sceue  and  create  a left-right  stereo  pair  froa  a 
single  iaage.  The  pcograa  is  guite  general  and  the  height  inforaation 
can  be  obtained  froa  a priori  knowledge  of  the  scene  height,  or 
artificially  created  froa  iaage  features  as  described.  The  height 
inforaation  is  supplied  as  a non-negative  nuaber  which  proportionately 
controls  the  perceived  height  in  the  left-right  stereo  pair.  For  the 
left  iaage  of  the  pair,  the  prograa  locally  shifts  pixels  to  the  right 
depending  on  the  heigat  inforaation.  In  the  local  shift  operation 
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certain  pixels  ace  obliterated  by  the  shift  and  other  surrounding 
pixels  aust  be  aoved  in  to  fill  in  the  aissing  area  of  the  shifted 
pixels.  The  prograi  iccoaplishes  this  autoaatically  and  efficiently 
xeeps  only  the  needed  iaage  line  in  core.  A scale  factor  can  be  also 
chosen  to  control  the  gain  on  the  perceived  depth.  For  the  right 
iaage,  the  operation  is  sisilar  except  that  the  local  pixel  shift  is 
to  the  lift.  The  scale  factor  is  also  used  with  unnorsalized  height 
data  to  control  the  aaxiaua  pixel  shift.  For  input  iaages  and 
artificial  stereo  pairs  of  256  x 256,  a aaxiaua  local  shift  of  10 
pixels  is  all  that  is  needed  for  an  exaggerated  depth  illusion. 

It  is  iaportant  tnat  the  input  infornation  be  locally  saoothed. 
Unsaoothed  height  data  produces  jagged  edges  on  objects  and  severe 
distortions  of  iaage  shapes.  Although  soae  aaount  of  distortion  in 
each  picture  is  tolarable  because  the  eye  tends  to  fuse  iaages,  any 
severe  distortion  detracts  froa  the  stereo  illusion.  It  should  also 
be  Mentioned  that  local  features  such  as  objects,  spots,  lines,  etc. 
aust  be  preserved  siaply  for  the  eye  to  follow  in  perceiving  depth. 
If  a uniforaly  featureless  flat  field  were  locally  shifted,  there 
would  be  no  perceived  depth  because  there  are  no  features  for  the  eye 
to  place  in  register. 

An  experiment  to  test  these  ideas  have  been  perforaed.  A Roberts 
gradient  edge  detector  was  used  to  create  the  height  inforaation. 
Denoting  an  iaage  array  by  f(i,j),  the  Roberts  operator  produces  an 
edge  nap 
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e(i,  j)  = l f(i,  j)  - f(i+l,j+l)|+  |f(i+l,j)  - f(j+l,i)|  (3.8-1) 

by  operatiug  in  a 2x2  array  of  pixels.  Tne  Roberts  operator  is  very 
affective  at  locally  isolating  edges,  but  using  the  edge  output  e(i,j) 
directly  for  height  inEoraation  produced  poor  results.  The  Roberts 
infornation  was  sioothed  using  a sinple  unifornly  weighted 
convolutional  averaging  window.  Various  window  sizes  have  been  tried, 
but  a window  of  3 pixels  horizontally  and  5 pixels  vertically  seeas  to 
work  well.  The  best  averaging  window  probably  depends  on  the  anount 
of  local  detail  in  tha  input  iaage. 

The  saoothed  Roberts  edge  data  was  used  as  the  input  to  the 
artificial  stereo  prograa  for  an  exper inenta 1 exaaple  using  a noon 
picture.  The  left-rigat  stereo  pair  is  shown  in  figure  1.  These 
pictures  are  best  viewed  with  a binocular  viewer  or  with  a prisu  held 
close  to  one  eye  so  that  the  iaage  will  fuse.  The  stereo  inpression 
iron  these  pictures  is  very  striking;  particularly  in  the  lower  center 
of  the  iaage  where  tha  edge  infornation  froa  the  craters  aakes  then 
appear  to  be  floating  at  different  heights. 

This  experiaent  is  only  preliainary,  and  considerable  further 
work  is  suggested  by  these  results.  Different  preprocessing  of  height 
infornation  is  a possibility;  Julesz  <1>  has  shown  that  stereo 
perception  with  left-right  pairs  is  reaarkably  unaffected  by  blurring, 
snail  size  reductions  or  noise  in  one  iaage. 
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4.  Beal  Tiae  Inpleaentation  of  Iaage  Processing  Technigues 

In  the  past  several  fears  a nuaber  of  device  concepts  and 
technologies,  such  as  charge  coupled  devices  (CCD)  and  integrated 
injection  logic  (I  U » save  been  developed  which  are  already  having 
significant  inpact  on  signal  processing  capabilities.  The  purpose  of 
this  task  is  to  study  aeans  of  perforaing  proven  iaage  processing 
algorithas  with  sues  technologies  in  integrated  circuit  fora.  The 
benefits  of  perforaing  the  processing  in  silicon  integrated  circuits, 
which  night  be  incorporated  into  an  iaage  sensor,  for  exaaple,  are 
clear  in  terns  of  powar,  weight  and  cost.  Tha  advantages  of  both  CCDs 
and  I L in  terns  of  taeir  power-delay  product  aake  then  particularly 
attractive,  as  shown  in  figure  1.  However,  it  is  not  intended  that 
the  study  be  United  to  these  two  technologies.  If,  for  exaaple,  the 
speed  of  Saks  PET  logic  can  add  significantly  to  the  capabilities  of 
the  processor  that  technology  will  be  included  and  the  cost  and 
benefits  exaained. 


Since  this  is  a new  task  soae  discussion  of  the  prograa  goals  and 
schedules  is  warranted.  The  inherent  flexibility  of  the  device 
technologies  under  consideration  iaplies  that  a very  large  nuaber  of 
algorithas  night  usefniLy  be  studied.  However,  at  the  present  tine  it 
ls  our  intention  to  expaasize  the  following  five  classical  operations: 


i 


(a)  Convolutional  processing  (both  spatial  and  via  chirp 
transf oraation  technigues) 

(b)  Edge  detection  via  the  Roberts  nagnitude  operator 

(c)  Edge  detection  via  the  Sobel  operator 


(i)  E 1 i e iletectioa  via  the  Haeckel  operator 
(e)  Successive  histogras  operations. 
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Each  of  these  operations  is  described  in  the  literature  and  only 
such  aspects  of  the  algorithms  which  specially  affect  the  circuit 
perforaance  will  be  discussed.  Within  each  topic  the  critical  circuit 
functions  will  be  identified  and  factors  affecting  the  accuracy,  speed 
and  power  reguiremnnts  are  analyzed.  The  aspects  of  the  device 
processing  which  iapact  the  circuit  perforaance  will  be  addressed. 
Where  appropriate,  discussion  of  the  processing  Units  will  be 
included.  For  those  operations  which  can  be  performed  with  present 
technology  a scheaatic  of  the  reguired  circuit  will  be  developed  and  a 
perforaance  analysis  undertaken.  Where  it  is  estinated  that 
significant  development  will  be  reguired  before  the  necessary 
circuitry  can  be  inpleaanted  the  critical  areas  will  be  identified. 

An  important  part  of  any  such  endeavor,  of  course,  is  a detailed 
review  of  the  state  of  the  art  of  both  the  device  technology  and  the 
processing  capabilities.  In  the  first  three  aonths  of  the  prograa, 
which  this  report  covers,  *e  have  undertaken  such  a review,  primarily 
of  the  charge  transfer  technology  and  its  application  to  chirp 
transform  processing.  A summary  of  this  work  is  included  here. 

During  the  next  quarter  we  intend  to  extend  the  analysis  of  CCDs 
to  include  the  other  four  functions  described  above  and  provide  block 

schematic  diagrams  and  circuit  layouts  where  possible,  prior  to 

2 

analysis  of  possible  £ L implementations. 


Page  203 


Another  objective  of  this  task  is  to  experimentally  study  the 
teal  tiae  implementation  of  nonlinear  optical  data  processing  (ODP) . 
In  this  part  of  the  tori,  we  are  utilizing  a liquid  crystal  light 
valve  as  an  incoherent  light  converter.  The  input  image  is  modulated 
by  a grating  pattern.  Tnis  process  is  the  real  tiae  implementation  of 
the  nonlinear  ODP  work  of  Dashiell  and  Sawchuk.  The  goals  of  this 
work  are  to  measure  tns  light  valve  transfer  function,  maximize  the 
MgnaaaN  of  this  relationship  and  implement  various  aonotonic  and 
nonmonotonic  nonlinear  functions.  In  the  first  quarter,  we  measured 
the  light  valve  transfer  function  with  different  drive  conditions  and 
demonstrated  a nonmonotonic  nonlinearity.  This  work  is  reported  in 
Section  4.3 

4.1  CCD  Technology  Review 

4.1.1  Charge  Transfer  Devices  Performance 

CCD  Device  Operation:  The  charge  coupled  device  (CCD)  concept 
<1,2>  consists  of  a linear  array  of  HOS  capacitors  with  input  and 
output  semiconductor  junctions  to  inject  and  extract  electrical  charge 
as  shown  in  figure  2.  Bultiphase  clocks  are  used  to  shift  the  charge 
from  one  capacitor  to  the  next.  Three  basic  types  of  CCDs  exist; 
surface  channel  (SC20)  , buried  channel  (BCCD) , and  aonuniformly  doped 
devices  such  as  the  peristaltic  (PCCD).  In  the  surface  channel  device 
the  HOS  capacitor  is  formed  by  a thin  (typically  **■  O.lu  ■)  silicon 
dioxide  layer  forsed  on  a silicon  substrate.  The  control  gates, 
usually  of  aluminua  or  heavily  doped  polysilicon  are  eposited  on  the 
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Figure  4. 1-2.  Surface  channel  CCD  and  potential  profile  of  SCCD. 
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Si02  surface.  Typical  gaoaetries,  using  photolithography,  have  gates 
with  lengths  of  10  to  25  ps. 

In  SCCD  devices  the  charge  is  transferred  directly  at  the 
silicon-silicon  dioxide  interface  where  i potential  aininun  exists. 
In  the  buried  channel  device,  a thin  layer  at  the  substrate  surface  is 
loped  so  that  the  potential  aininun  is  located  a snail  distance  fron 
the  silicon-silicon  dioxide  interface.  In  this  case  the  charges 
travel  in  the  substrate  and  hence  aany  of  the  trapping  states  are 
avoided,  yielding  a higher  charge  transfer  efficiency  (CTB) . A 
nchenatic  of  a buried  channel  device  is  shown  in  figure  3. 

Peristaltic  <3>  devices  rely  on  varying  the  doping  density  within 
the  buried  channel  to  achieve  high  charge  storage  (as  in  surface 
channel  operation)  and  also  achieve  high  CTB  by  transferring  the  last 
fraction  of  charge  within  the  substrate  by  field-aided  drift  (as  in 
the  buried  channel) . Devices  using  this  technique  have  been  operated 
at  frequencies  above  100  HHz,  and  further  increase  in  speed  depends 
largely  on  the  developaant  of  high  speed  input  and  charge  sensing 
devices.  A comparison  of  the  bandwidth  and  processing  delay  available 
with  the3e  devices  as  coepared  with  surface  wave  acoustic,  is  shown  in 
figure  4. 

The  operation  of  charge  transfer  devices  has  been  described 
elsewhere  <4,5>,  but  for  the  sake  of  coapieteness  the  concept  is 
reviewed  briefly  here. 


Figure  2 shows  a single  cell  of  a typical  p-type  surface  channel 


(a)  Buried  channel  CCD  (BCCD) 


(b)  Potential  profile  in  BCCD 


Figure  4.  1-3.  Buried  channel  CCD. 
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device.  A positive  aiactrode  voltage  applied  to  the  device  repels  the 
aajority  carriers  creating  a depletion  region  directly  beneath  the 
gate.  The  energy  hands  in  the  seaicondnctor  then  becoee  bent  and  a 
potential  ainiaun  exists  directly  at  the  silicon- silicon  dioxide 
interface.  If  charge  is  now  injected  into  the  device,  in  the  fora  of 
aajority  carriers,  in  this  case  electrons,  they  will  be  stored  at  the 
interface  aodifying  tae  potential  profiles  as  shown  in  figure  2b.  The 
aaxiaua  charge  density  Mmax  can  be  written  in  terns  of  the  gate 
voltage  ?G  as 


N 


max 


7T [VG  ' VFB  - * Vb] 


(4.1-1) 


where  ?FBis  the  flat  band  voltage,  $ is  the  well  potential  a 


nd 


VB=-C 


( 4eN  i ) 
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ox 
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>IvFB  " 2\  " VB 


(4.  1-2) 


and  hence  the  charge  storage  is  directly  proportional  to  the  gate 
voltage,  i.e. 


N A 

max  e 


(4.1-3) 


where  A is  the  charge  storage  area. 

The  charge  transfer  is  effected  by  clocking  adjacent  gates  so  as 
to  create  increasing  well  depth  in  the  direction  of  desired  charge 
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travel.  Numerous  clocking  techniques  have  been  eaployed  ranging  f roa 
a singla  cycle  to  fall  four-phase  clocking.  The  stepped  oxide 
technique  is  coanonly  eaployed  to  achieve  two-phase  operation  where 
two  adjacent  gates  are  connected  together  and  the  variation  in  oxide 
thickness  provides  the  different  well  depths. 


Por  SCCD  devices  the  charge  travels  at  the  silicon  Sio2  interface 

where  the  density  of  trapping  states  is  high,  typically  of  the  order 
9 -2  - 1 

of  10  ca  eV  . The  ZIE  is  therefore  necessarily  lower  than  for  BCCD. 
dowever,  CTE  better  taan  0.999  have  been  obtained  with  SCCD  devices  by 
using  a "fat  zero"  whica  involves  constantly  clocking  through  a saall 
background  charge.  The  ratio  of  the  charge  reaaining  N t(t ) after 
tiae,  to  the  original  charge,  Hq(  in  the  absence  of  field  aid  drift  is 
given  by 


* e V \ 

2L2C  ) 


(4.  1-4) 


where  un  is  the  electcon  nobility,  L is  the  distance  between  adjacent 
gates,  and  C is  the  effactive  capacitance  per  unit  area. 


If  fringing  fields  exist  as  in  the  BCCD  the  relationship  becoaes 


•xp 


-3L3C 


(4.  1-5) 


where  A 7 is  the  voltage  difference  between  the  initial  and  final 
sites.  An  analysis  of  the  relationship  between  these  two  aechanisas 
shows  that  the  field-induced  drift  is  dosinant  in  the  final  stage  of 
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the  charge  transfer  and  is  essential  for  high  CTE  at  high  freguencies. 
Since  most  of  the  electrostatic  field  is  normal  to  the  direction  of 
cnarge  transfer,  "complete"  charge  transfer  can  be  accomplished  only 
at  relatively  low  clock  rates  (of  the  order  of  103  kHz) . In  the 
ouried  channel  concept,  a layer  of  n~doped  Material  is  introduced  on 
the  surface  of  the  p-typ*  silicon  by  epitaxial  or  ion  iaplantation. 
When  a positive  voltage  is  applied,  a depletion  region  is  formed  under 


the  electrode  and  the  resulting  potential  profile  is  as  shown  in 
figure  Jb.  The  potantial  minim*  $min  is  determined  by  the  gate 
voltage  V„  as 

U 
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This  function  is  plotted  in  figure  5,  (which  is  taken  fro*  <5>) 
showing  that  the  potential  is  approxiaat-aly  a linear  function  of  ?G 
over  a wide  range  of  gate  voltages.  When  charge  is  injected  into  the 
device  it  resides  at  the  potential  ainiaua  which  is  typically  several 
nundred  angstroas  froa  the  interface.  This  allows  aore  effective 
charge  transfer  and  wTB  as  high  as  0.9999.  Purther  the  nonlinear  or 
transverse  fields  are  considerably  larger  at  these  depths  and  account 
for  a considerable  part  of  the  charge  transfer.  This  in  turn  allows 
the  higher  speeds  to  oe  achieved,  as  illustrated  in  figure  2. 

The  peristaltic  concept  coabines  the  high  charge  storage 
capabilities  of  the  SZCD  with  high  charge  transfer  efficiency  and 
large  bandwidth  capabilities  of  the  BCCD.  It  achieves  this  basically 
by  storing  the  charge  at  the  surface  of  the  silicon  while  transferring 
the  last  charges  deep  in  the  silicon  substrate  where  the  fringing 
fields  are  strongest.  Of  the  three  transfer  aechanisas,  self-induced 
drift,  tharaal  diffusion  and  fiald  aided  drift  the  foraer  accounts  for 
90  to  99%  of  the  transfer  and  is  accoaplished  in  a few  nanoseconds. 
The  theraal  diffusion  is  usually  such  slower  and  is  significant  only 
in  the  final  stagas  of  the  SCCD  transfer  process.  Peristaltic 
operation  stores  the  caarge  close  to  the  silicon  surface  so  as  to 
achieve  high  storage  as  in  the  SCCD  but  the  doping  profile  is  aodified 
so  as  to  force  the  last  fraction  of  charge  into  the  bulk  during  the 
transfer  process  whsra  field  aided  drift  accounts  for  the  final 
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transfer. 

input  Circuitry:  Tab  spend  and  efficiency  of  tha  CCD  input  and 
output  circuitry  is,  of  course,  an  important  consideration  in  the 
ultimate  performance  of  the  final  device.  Tvo  basic  concepts  exist  to 
convert  the  input  voltage  signal  into  the  required  charge  for  the  CCD. 
They  involve  either  integrating  a current  for  a fixed  tiae  interval  or 
charging  a fixed  capacitor.  The  integrating  input  requires  less 
peripheral  circuitry,  provides  voltage  gain,  anl  generally  has  lower 
noise.  The  capacitor  input  (Toapsett  input  or  threshold  correcting 
input)  is  aore  linear  and  is  less  sensitive  to  bias  conditions.  Both 
inputs  have  approxisately  the  sane  transfer  characteristics. 

k schematic  of  the  integrating  input  is  shown  ia  figure  6.  The 
input  signal  is  ac  coupled  to  the  TIN  gate,  thus  eodulating  the  asount 
of  current  trickling  into  the  CCD.  For  this  particular  input,  charge 

a 

continues  to  trickle  regardless  of  the  state  of  . The  anount  of 
charge  finally  collects!  in  the  *.  transfer  bucket  is,  therefore, 
proportional  to  the  integral  of  the  input  voltage  over  the  full  clock 
period.  The  input  is  vary  sensitive  to  the  bias  voltages  at  the 
source  iiffusion  anl  the  input  electrode,  and  becoaes  increasingly 
nonlinear  for  large  ac  values  of  ?.  due  to  the  input  ROSPET*s  square 
law  dependence  of  current  on  gate- to** source  voltage.  The  sensitivity 
problem  of  this  input  is  further  heightened  by  the  traditionally 
unpredictable  threshold  voltage. 


The  threshold  correcting  input,  shown  in  figura  7 solves  nany  of 


the  problems  at  a cost  o£  gain  and  sane  noise.  The  inpnt  diffusion  is 
driven  by  $.  to  a level  where  the  gate  is  overflowed.  The 
diffusion  then  recedes  and  the  extra  charge  fails  back  with  it  leaving 
an  anount  of  charge  proportional  to  the  difference  between  V.n  and 
Vdc‘  *1  fails  to  receive  the  input  charge,  it  picks  up  an  anount 
of  charge  proportional  to  the  nininun  of  the  input  signal  during  the 
input  aperture  tine. 

The  threshold  correcting  input  nakes  use  of  the  fact  that 
although  the  threshold  voltage  nay  vary  across  the  chip  and  fron  wafer 
to  wafer,  the  variation  between  two  adjacent  (or  alaost  adjacent) 
gates  can  be  nade  vecy  snail  thus  ensuring  that  when  the  V-n  gate  and 
the  Vdc  gate  are  at  the  sane  potential,  no  charge  will  be  entering  the 
CCD.  This  differential  aspect  of  the  input  scdene,  conbined  with  the 
large  input  full  scale  signal,  sake  the  input  biasing  very 
noncriticai.  This  voltage  type  of  input  is  also  independent  of  clock 
rate  unlike  the  previous  integrating  input.  The  speed  of  input 
.devices  of  this  type  is  presently  United  to  about  100  MHz  but  high 
resolution  bipolar  circuitry  presently  being  developed  nay  increase 
this  to  several  hundred  MHz. 

Output  Circuitry:  It  the  output  of  the  CCD  the  charge  nust  be 
converted  into  a voltage.  This  can  be  achieved  by  discharging  the 
signal  through  a precision  controlled  resistor  and  Measuring  the 
rHsulting  voltage.  Alternatively  the  charge  can  be  dueped  into  a 
capacitor  and  the  voltage  aeasured. 
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Figure  4.  1-7.  Threshold  correcting  input. 
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The  foraer  technique  is  illustrated  in  figure  8.  It  is  basically 
siapler  to  iaplenent  and  operates  at  high  speeds.  In  general,  when  B 
is  large  enough  to  aatcn  the  resistor  noise  current  to  the  CCD  noise 
current,  the  signal  bandwidth  is  below  one-half  the  clock,  frequency. 
To  increase  the  bandwidth,  a transinpedance  amplifier  is  used  instead 
of  the  resistor.  The  transiapedance  aaplifier  is  generally  an  op  aap 
with  a feedback  resistor  B.  The  output  diffusion  is  tied  to  the 
sunning  junction  of  the  op  aap,  and  the  output  voltage  bias  is 
connected  to  the  positive  input  of  the  aaplifier.  Such  a circuit  is 
shown  in  figure  9.  The  advantage  of  this  low  input  iapedance 
transiapedance  aaplifier  is  that  the  effect  of  the  parasitic 
capacitance  on  the  output  node  is  divided  by  the  open  loop  gain  of  the 
aaplifier.  Discrete  circuits  capable  of  handling  10  pP  on  the  output 
node  have  been  built  with  a transiapedance  of  53  K ohas  at  signal 
frequencies  up  to  10  BBz.  The  bandwidth  can  be  extended  further  if 
lower  transiapedances  are  acceptable. 

The  second  outpat  circuit  (norailly  called  a reset  output)  is 
aore  coaaon  than  the  direct  current  node  type.  The  two  kinds  of  reset 
output  circuits  are  called  floating  diffusion  and  floating  gate.  k 
scheaatic  of  the  floating  diffusion  output  is  shown  in  figure  10.  The 
aethod  of  operation  for  this  output  is  (1)  while  (JHST  is  high,  0 2 
rises  to  duap  charge  over  the  screen  gate;  (2)  this  charge  increases 
the  surface  potential  around  the  floating  diffusion  thus  increasing 
the  diffusion  potential;  and  (3)  the  sourca  follower  rises  as  the 
floating  diffusion  rises.  The  output  is  reset  when  0RST  lowers  and 
allows  the  signal  charge  to  flow  into  the  VDD  supply.  The  output 
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Figure  4. 1-9.  DC  output  using  transimpedance  amplifier. 
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voltage  froa  this  circuit  is  constant  with  clock  frequency  if  the  duty 
cycle  of  the  output  is  held  constant.  The  aagoitude  of  the  output  is 
determined  by  the  asount  of  full  scale  charge  and  the  capacitance  on 
the  output  node.  Vita  a 50*  duty  cycle,  the  output  aagnitude  can  be 
as  high  as  0.5  V res. 


Using  standard  peotolithography  output  circuits  of  this  type  have 
been  operated  routinely  at  20  NHz.  Presently  these  bipolar  devices 
are  being  developed  usiag  electron  bean  exposure  to  operate  at  speeds 
in  excess  of  100  NHz. 


Charge  Transfer  Efficiency  (CTE) : The  charge  transfer  process, 
even  for  a single  transfer,  is  never  conplete.  A certain  fraction  of 
charge,  e , is  lost.  Tna  principal  causes  of  the  incoaplete  transfer 
are  the  finite  velocity  of  the  carriers  and  the  trapping  states.  The 
CTE  is  usually  lower  for  SCC0  devices  than  for  BCCD  because  (a)  the 
density  of  trapping  states  are  greatast  at  the  Si-SiO^  interface  and 
(b)  the  transfer  Is  usually  inconplete  (at  all  but  the  very  slowest 
clock  rates)  since  field  aided  drift  is  aosent.  In  addition  the 
charges  trapped  in  the  interface  states  espty  slowly  and  hence 
contribute  to  trailing  data.  In  fat  zero  operation  a constant  charge 
level,  of  10  to  20%  of  the  fall  signal  level,  is  used  to  keep  all  the 
states  full  and  than  reduce  the  charge  loss.  The  technique  is 

-4 

primarily  used  for  digital  applications  and  can  result  in  10 


The  density  of  traps  deep  in  the  silicon  is  auch  less  and  hence 
the  need  for  fat  zaco  in  BCCD  is  reduced.  A principle  effect  of  the 
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tinite  charge  inef fLciancy,  apart  froe  resulting  attenuation  and 
Uniting  the  dynaaic  range,  is  to  liait  the  total  nuaber  of  transfers 
possible.  Since  the  charge  attenuation  is  cumulative,  the  overall 
characteristic  for  n transfers  is  given  by 


= (1  - O = 1 - nc 


(4.  1-7) 


To  avoid  excessive  lagcadation  in  signal  to  noise  ratio  usually 
requires  less  thaiKO.  1/e. 

The  effect  of  charge  spilling  into  trailing  data  slots  is  to 
create  dispersion.  This  can  be  seen  froa  the  Z-transforn 
characteristic  of  a single  stage,  which  can  be  written  as 


T(Z)  = 


■ !■  '• 


where 


(4.  1-8) 


Z = exp 


Pr] 


(4.  1-9) 


and  f£  is  the  applied  clock,  rate. 


For  a transversal  filter  with  ideal  inpulse  respanse 


t<z>  = £ V 


(4.  1-10) 


where  h.  are  the  weigating  coefficients,  the  actual  transfora  can  be 


written  as 
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(4.1-11) 


It  has  baea  shown  that  this  is  aguivalent  to  the  transfer  function  of 
an  ideal  device  evaluated  at  frequency 


rl  = 


= f + 


£ *in(T?)  -jj‘  -c°*(«r)i 


(4.1-12) 


which  represents  a frequency  shift,  or  dispersion,  given  by 


Af  = 


(4.  1-13) 


and  an  envelope  nodulation  given  by 

A(f)  = 1 


*[‘  - CO.  (4*) 


(4.  1-14) 


The  effect  of  this  on  the  nodulation  transfer  function  is  shown 
in  figure  11.  For  large  fractional  bandwidth  the  product,  ne.  nust  be 
•aintainad  as  saall  as  possible.  Thus  for  a large  tine-bandwidth 
product,  the  signal,  e . nust  be  reduced  as  auch  as  possible.  Further 
it  can  ba  shown  that  the  charge  transfer  inefficiency  does  not  alter 
the  signal-to-noise  ratio  to  first-order. 


Accuracy  of  Waigating  Technigue:  Typically  -CD  devices  are 
weighted  by  the  split  electrode  technique  illustrated  in  figure  12. 
As  shown,  both  ends  of  each  storage  electrode  are  connected  to  a 
differential  current  aaplifier  which  senses  the  charge  directly 
beneath  it.  The  location  of  a saall  gap  deteraines  the  tap  weighting. 


A 


Foe  exaaple,  if  to  i gates  have  length  L and  the  distance  between 
the  gap  and  the  center  of  the  gate  is  h^  the  fraction  of  charge 
coupled  to  the  positive  and  negative  clock  line  is  given  by 

V • t'1+V 

and 

Qk’  = T (1  ‘V  (4.1-15) 

respectively.  Then  tae  output  signal  is 

vo<'>  " Z [AV  <*>  - av<'>1 

k L J 

or 

Vo(t)  = const  ^2  ^V.^t-kT)  (4.1-16) 

k 

Weighting  errors  can  tharefore  be  caused  by  inaccuracies  in  the  Bask 
faorication  or  alignaant  as  well  as  nonunif orsities  in  the  oxide 
thickness  and  substrate  sensitivity.  Of  these  the  principle  source  of 
coherent  error  is  the  guantization  error  introduced  in  the  sask 
generator.  The  actual  resolution  attainable  after  exposure  and 
etching  is  usually  considerably  less  than  that  of  the  eask.  For 
exanple  routine  photolithographic  techniques  are  liaited  in  resolution 
to  about  Sui.  For  a gate  length  of  200  u ■ this  represents  an  error 
of  about  The  situation  is  such  improved  if  electron  bean 

■icrofabr ication  teennigues  are  eaployed.  For  exaaple  the  facilities 
developed  at  Hughes  Rasaarch  Laboratories  use  a 15  bit  D/A  converter 
to  provide  resolution  of  1 part  in  32*000,  and  can  provide  final  line 
accuracies  of  about  0,2  u a in  an  overall  scan  of  1 as.  Hence,  for 
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typical  gate  lengtas  of  200  ua  an  error  of  approxiaately  0.05%  is 
attainable. 

The  effect  of  weighting  errors,  cr^  , on  the  overall  HTF  of  a 
transversal  filter  can  be  written  in  the  terns  of  eg. (10)  as 


?<f>  = £ <hk  + ,k,  L, 


where  T (f ) is  the  distorted  transfer  function.  The  error  teas 


AT(f)  = T(f)  - T(f) 


then  can  be  expressed  as 


AT(f)  = 


c 


In  general  the  exact  evaluation  of  AT(f)  depends  on  the  desired  filter 

response  T(f)  and  the  characteristic  of  the  weighting  errors. 

however,  if  the  weigating  errors  are  assuaei  to  be  randoa  with 

variance  a then  the  fractional  error  can  be  written  as 
rms 

ATM  = --°rms  /a 


where  U is  the  channel  width. 


For  a device  fabricated  using  photolithography  and  with  100 
3tages,  H = 200u  a and  aTTns  * 2ua 


- q 2% 

T(f) 


/ 
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With  elactcon  bean  fabrication  techniques  this  error  can  be  farther 
i educed  by  about  an  order  of  nagnitude. 

Factors  Affect  in  3 :::d  Noise:  Noise  is  an  important  consideration 
in  all  CCD  applications  and  novel  circuit  technigaes  have  been 
developed  to  ainiuize  its  effects.  Three  basic  noisa  sources  can  be 
identified  as  (1)  theraal  noise  in  the  input  and  output  circuitry,  (2) 
noise  due  to  trapping  states,  and  (3)  dark  currant  noise.  A nuaber  of 
low-noise  circuits  aave  been  developed  to  ainiaize  the  noise  in  the 
input  and  output  circuitry.  Typical  of  these  are  the  correlated 
double  saapling  output  circuit  and  the  distributed  gate  aaplifier. 
For  specific  details  of  these  circuits  the  reader  is  referred  to 
references  <4>  and  <5>. 

Since  the  CTE  is  liaited,  a fraction  of  tha  charge  is  left  behind 
in  each  transfer  and  soae  additional  charge  is  gathered  froa  the 
preceding  data.  In  general  the  signal  fluctuation  depends  on  the  data 
foraat  but  an  effactive  noise  signal  can  be  evaluated  assualng 
independent  fluctuations  as 

AQ  = 2n  AQtr2 

2 

vhereAQtr  is  the  naan  sguare  fluctuation  in  each  transfer.  The 
noise  spectral  density  Str  (f)  has  been  calculated  as 

v<>  (if)] 

for  low  transfer  inefficiency,  i.e. , e <<  i/q  showing  the  noise  at 
high  frequencies  to  be  enhanced. 
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A farther  source  of  noise  is  associated  with  the  statistical 
emission  of  carriers,  fhornber  <6>  has  shown  that  this  varies  between 


t KTC 
y s 


KTC  s 


for  different  parts  of  the  clock  cycle. 


The  trapping  states  within  the  SCCD  contribute  as  another  source 
of  noise,  since  charge  is  absorbed  and  eaitted  independently  of  the 
clock  cycle.  The  cnacge  fluctuation  fros  these  sources  can  be 
evaluated  as 


AQ..2  - I2  kT  A.  N„  1"  (2) 

and  hence  an  effective  signal-to-noise  ratio  for  a given  charge  packet 


can  be  calculated  as 

C A C 2 V 2 

— - a ox  a 

N ' 2n  kT  In  (2) 

9 -2  -1 

where  N8S  is  the  interface  trap  density  (typicaLly  ~ 10  ca  eV  ) . 
Siailar  noise  contributions  are  nade  in  BCCD  due  to  the  bulk  traps  but 
in  this  case  the  noise  is  frequently  dependent,  reaching  a naxisun 
when  the  transfer  tiae  is  approxinately  equal  to  the  emission  tiae 
constant.  Effective  aodels  for  the  noise  froa  bulk  traps  have  not 
been  developed.  For  lost  BCCD  devices  the  aguivalent  noise  phenomenon 
is  perhaps  an  order  ot  magnitude  less  in  BCCD  than  SCCD,  and  is 
usually  dominated  by  input  and  output  noise  sources,  k full  analysis 


Addition;  The  basi;  serial  operation  of  a CCD  shift  register 
lakes  it  a natural  technology  for  analog  sampled  data  operations. 
Linearity  of  the  ordac  of  1%  can  be  achieved  over  tiee  bandwidth 

3 

products  of  about  Id  ; the  data  being  stored  in  the  fore  of  finite 
aaounts  of  charge.  Addition  can  nost  easily  be  acconplished  by  sieply 
aerging  the  charge  fcoe  two  or  sore  potential  wells  into  a cell 
bucket.  The  technique  is  illustrated  in  figure  13  where  the  contents 
of  two  parallel  analog  delay  lines  are  dunped  into  a single  line.  In 
this  case  the  voltage  on  a screen  separating  the  two  channels  is 
pulsed  to  allow  charge  flow.  The  technique  is  not  linited  to  any 
uuaber  of  separate  lines  and  can  be  used  to  fora  a multiplexer, 
further,  by  altering  the  well  size  (by  varying  the  area  of  the 
receiving  gate)  the  resultant  output  can  be  scaled  by  a constant 
factor. 
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Generalized  subtraction  is  a soaewhat  sore  coaplex  operation  and 
is  not  naturally  suited  to  CCD  technology.  One  technique,  for 
exanple,  night  involva  Corning  a bucket  eith  capacity  equal  to  the 
saaller  of  the  two  charges  in  the  operation  and  allowing  charge  to 
irain  into  this.  Onf octunately  this  requires  HOS  transistors  and 
capacitors  and  is  in  general  not  an  effective  technique. 


Multiplication;  Sultiplication  can  be  perforaed  in  CCD  coapatible 
technology  by  using  MOSPETs  and  varying  both  the  signal  to  the  source 
and  gate  siaultaneously  <8>.  The  technique  involves  using  the  FETs  as 
voltage-controlled  resistors.  Further,  since  higher-order  nonlinear 
products  are  available,  distortion  terns  can  be  generated  and  used  to 
null  out  unwanted  ter  13  in  the  output.  Osing  these  techniques  devices 
have  been  operated  at  about  100  kHz  with  accuracy  of  IX.  By 
increasing  the  resolution  of  the  circuit  this  aignt  be  inproved  to 
about  0.5X  at  1 MHz. 


An  alternative  technique  <9>  which  eaploys  the  CCD  structure 
itself  to  provide  aultiplication  relies  on  the  aultiplying  D/A  concept 
illustrated  in  figure  1h.  Here  a binary  encoded  work  equivalent  to 
one  input  is  used  to  switch  in  and  out  parallel  CCD  gates  and  adjust 
the  devices  capacitance.  In  this  way  the  capacitance  is  nade 
proportional  to  one  input  and  the  driving  voltage  applied  across  the 
array  proportional  to  tne  second.  The  output  charge  is  hence  given  by 


Q 

F 


i 1 

i 

11 


and  is  proportional  to  the  product  of  the  two  signals 


At  present 
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aevices  with  8 bit  accuracy  have  been  operated  at  speeds  up  to 
100  xHz.  Two  inherent  liaitations  exist  in  this  type  of  circuit. 
First,  on.;  input  has  to  be  A/0  converted  and  hence  it  is  a nixed  aode 
operation.  This  coull  nowever  be  advantageous  in  the  chirp  transfora 
tecunique  where  the  nalti plying  chirps  aight  be  stored  in  digital  fora 
in  a HCN.  Since  all  ua  capacitors  are  filled  in  parallel  the  device 

is  not  inherently  slow,  but  it  does  require  a lot  of  "real  estate." 

3 

For  exanplo,  in  a 10-oit  converter  the  capacitor  range  is  1:10  . Por 
this  reason,  by  using  photolithography,  the  technique  is  liaited  to 
about  8 bit  accuracy.  However,  by  increasing  the  resolution  with 

electron  bean  exposure  the  saallest  capacitance  aight  have  an  area  of 

2 

lua  and  11  bit  accuracy  aight  be  achieved.  This  would  enable  the 
chirp  transfora  to  ba  iapleaented  in  aonolithic  fora,  as  discussed  in 
Section  Ill-A.  At  presant  5 MHz  operation  is  considered  possible  with 
this  technique  using  elactron  beaa  exposure. 

Transverse  Piltarino:  The  CCD  structure  with  serial  delay  and 
parallel  output  via  the  split  electrode  technique  is  a natural 
coaponent  for  transversal  filtering.  At  present  devices  with  an 
iapulse  response  of  1 sec  have  been  constructed  with  20  to  50  output 
taps.  As  discussed  in  Section  II-A-5,  the  weighting  accuracy  is 
currently  0.1  to  0.05ft.  Qy  eaploying  correctly  apodized  band-pass 
filters,  fixed  aatchad  filters  or  correlators  can  be  iapleaented 
lirectly. 

An  iaportant  area  which  is  receiving  interest  at  the  present  tine 
is  the  possibility  of  developing  prograaaable  filters  by  building 
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variable  nondestructive  tapping  circuits.  The  concept  of  the  floating 
clock  electrode  sensor  (PCE5)  is  presently  being  developed.  By 
driving  a source  follower  the  charge  can  be  read  nonlestructively  and 
provide  a low  output  impedance  source.  The  circuitry  discussed  in 
reference  <10>  for  tais  is  shown  in  figure  15.  At  present  the 
capacitance  load  on  the  sensing  HOSFET  limits  the  dynamic  range,  but 
with  careful  design  and  using  a short  gate  bipolar  output  this  problem 
mignt  be  overcoaa.  Further,  by  using  voltage-controlled  FET 
conductance  the  technique  can  ba  fully  electronically  programmable. 
At  present  the  accuracy  is  limited  to  about  3X  at  103  kHz. 


An  electronically  programmable  correlator  (or  filter)  can  be 
implemented  by  using  two  counter-running  analog  CCD  delay  lines  and 
reeding  the  parallel  outputs  into  a series  of  HOSFET  multipliers  (as 
discussed  in  Section  III-B-2).  This  technique  requires 
synchronization  of  the  signal  carrying  the  correlation  (or 
convolution)  function  with  the  input  data  and  is  limited  to  at  most 
50X  duty  factor  for  all  but  sliding  convolution.  However,  this  may 
have  application  where  either  the  data  is  intermittent  or  the  phase 
information  is  not  required. 


A/D  Conversion:  Analog  to  digital  conversion  can  be  implemented 
directly  with  CCDs  oy  forming  potential  wells  equivalent  in  capacity 
to  each  quantized  bit,  and  moving  charge  through  the  array.  The 
circuit  required  is  siailar  to  that  shown  in  figure  14.  A comparitor 
circuit  must  be  included  at  each  site.  Again  the  technique  is  liaited 
to  about  8 to  10  bits  and  is  quite  bulky  in  this  implementation. 
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Techniques  to  reduce  the  area  of  the  silicoa  required  and  hence 
ieprove  tha  reliability  ate.  have  been  suggested  involving  successive 
approxiaations  and  coaparator  operations.  In  general  the 
CCD~iapleaented  techniques  are  presently  United  both  in  accuracy 
(nunber  of  bits)  and  speed. 


Digital  Logic:  A full  range  of  logic  functions  involving  only  CCD 
eleaeots  has  been  developed  <11>  and  auch  of  the  interest  in  CCD 
device  applications  is  centering  on  digital  iapleaentation,  because  of 
its  inherent  accuracy  and  the  ease  of  regeneration.  As  an  exaaple  of 
this  type  of  processing  two  functions,  the  OR/AND  gate  and  the 
EXCLUSIVE  OR,  are  described. 

A scheaatic  of  tae  layout  for  the  DR/AND  gate  is  shown  in  figure 
16a.  The  two  input  signals  are  fed  into  parallel  gates  A and  B with 
equal  capacity.  When  a clock  pulse  is  applied  to  the  transfer  gate 
the  contents  of  botn  channels  flow  into  toe  bucket  under  gate  C. 
Since  the  area  of  this  gate  is  identical  to  both  A or  D,  a single 
input  (or  one)  is  sufficient  to  fill  the  bucket  and  be  read  as  one. 
An  iaplanted  barrier  separates  this  bucket  froa  the  succeeding  one. 
In  the  event  of  bota  inputs  A and  B being  "ones'*  the  excess  charge 
spills  over  to  C and  is  read  as  a one,  representing  the  AND  output. 

The  accuracy  of  the  technique  is,  of  course,  extresely  high  since 
binary  decisions  only  are  required  and  frequent  refresh  operations  can 
be  perfocaed.  The  single  control  gate  is  the  only  active  elesent  and 
is  not  required  to  be  linear,  iiith  buried  channel  devices  the  speed 
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is  Halted  by  the  final  input  and  output  sensors. 

The  schematic  of  the  EXCLUSIVE  SB  is  shown  in  figure  16b.  Again 
the  two  inputs  are  fed  in  via  gates  A and  B,  and  the  operation  through 
gate  C is  as  described  move.  However,  the  charge  under  gate  C is 
sensed  and  connected  to  the  control  gate  separating  C and  E.  In  the 
advent  both  inputs  ara  "ones”  the  control  gate  naintains  a voltage 
barrier  over  which  the  charge  froa  gate  C is  prevented  froa  flowing. 
The  absence  of  charge  then  reads  as  a "zero."  If  only  one  input  is 
present  the  control  gate  will  not  aaintain  the  voltage  barrier  and  the 
charge  froa  C will  flow  into  the  final  gate  registering  as  a "one." 
The  performance  of  this  circuit  relies  on  the  effective  operation  of 
the  refresh  circuit  as  described  in  reference  <11 >.  This  requires 
that  the  floating  diffusion  accurately  following  the  charge  level 
under  gate  D.  This  requires  effective  decoupling  froa  the  surrounding 
tixed  gate  potentials.  In  practice  the  capacitive  coupling  to  the 
adjacent  gates  and  substrates  is  the  liaiting  factor  and  high  speed 
operation  depends  on  reducing  the  overlap  capacitance. 

h.2  Chirp  Transfora  I apleaentation  Using  CCOs 

The  chirp-Z  transfora  (CZT)  algorithm  provides  a technique  for 
performing  Fourier  tc ansf oraation , convolution  and  correlation  of 
aroitrary  signals  in  real  tiae.  It  basically  relies  on  restructuring 
tne  Fourier  integral  iito  two  aultiplications,  and  a linear  filtering 
operation  which  can  be  perforaed  with  a transversal  filter.  It  is  an 
iaportant  processing  technique  because  of  its  siaplicity  in  terns  of 
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Hardware  and  because  of  its  speed  of  operation.  The  aost  widely  used 
technique  for  Fourier  transf oraation  at  present  is  the  fast  Fourier 

transfora  (FFT) . It  requires  Hlog2(M)  coaplex  lultiplications  to  fora 

2 

an  H point  transfoca  as  contrasted  to  H operations  for  the  direct 
technique.  The  chirp  transfora  however  can  essentially  operate  in 
real  tine.  The  delay  involved  being  only  that  of  the  filtering 
operation. 

The  advent  of  CCD  transversal  filters  aakes  it  possible  to  fora 
3 4 

transforms  of  1C  and  13  points  with  speeds  as  high  as  100  MHz.  With 
ouried  channel  operation  the  basic  transversal  filter  can  operate  at 
frequencies  as  high  as  100  MHz  but  the  bipolar  sensing  devices 
required  to  interface  the  basic  CCD  structure  with  the  external 
circuitry  presently  liait  their  bandwidth  to  23  MHz  or  so.  It  should 
be  eaphasized  that  CCD  filters  are  operated  as  essentially  base-banded 
devices  and  hence  the  aaxiaua  operating  frequency  is  the  device 
bandwidth,  and  the  tiae  bandwidth  product  i3  the  nuaber  of  stages. 

The  basic  CZT  algorithm  derives  directly  froa  the  discrete  fora 
of  the  Fourier  transfora 


* v — * 

= J2  fn  **P 1 


.2 

•J  ttiiKt 
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where  the  data  to  be  transforaed  fn (0  < n < N - 1 ) can  be  considered  to 
be  unifora  tiae  saaplas  of  an  input  signal  f(t).  The  transforaed  data 
FK  then  represents  a uniforaly  saapled  version  of  tha  signal  spectrua 


F (Ui)  where 


LT3T  'A.lfWi. 
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FM  = /f(t)  dt 


(4.2-2) 


Bluestein  et.al.  <12>  recognized  that  by  expressing  the  coaplex 
exponent  of  eg. (1)  in  ta e fora 

2nk  = n2  + k2  - (n  - k)2 


(4.  2-3) 


the  integration  can  be  ceplaced  by  a Multiplication  and  convolution. 
This  can  be  seen  by  writing 


FK  = e 


-jnk2  N-l 


-Jim2  ju(k-n)2 

f e N e ^ 

n 


n=0 


(4.  2-4) 


which  is  the  fora  of  the  CZT.  Signals  of  the  fora 


N 


and 


exp[il^-} 

* N 

are  known  as  chirp  signals  froa  pulse  coapression  radar  <13>  and  are 
basically  signals  wiose  freguency  is  linearly  related  to  tiae.  The 
nardware  to  perforn  tae  algoritha  of  eg.  (4)  is  shown  in  figure  1.  The 
syabol  * represents  convolution,  in  this  figure,  with  a chirp 


such  that 
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(4.2-5) 


This  is  the  basic  operation  of  a chirp  filtar 

and 

hence 

can 

be 

inplanted  directly 

with  either  a surface 

acoustic 

wave 

or 

CCD 

transversal  filter. 

For  a CCD  inplaaentation 

the 

hardware 

is 

nore 

coupler  as  both  in  phase  and  quadrative  channels  are  required  as  shown 
iu  figure  2.  The  weigating  function  on  the  filters  are  basically 
sauples  of  the  linear  PH  or  chirp  functions 


h(nT) 

g(nT) 


cos  p(nT)2 
sin  p(nT)2 


< nT< 


Td 


(4.  2-6) 


where  u is  the  chirp  cate,  -Td  is  the  duration  of  the  filter  inpulse 
response,  and  T is  the  clock  period.  Due  to  the  finite  iapulse 
response  of  practical  filters  the  frequency  resolution  is  liaited. 
For  exaaple,  the  transforn  of  a single  CV  signal  sin(u>,t)  appears  at 
the  output  as 

6<u -*»!>•  sine  <|*Tdt) 


Presently  the  thernal  generation  of  carriers  within  the  CCD  cells 
Units  the  naxiaua  storage  tine  Td  to  about  1 sec,  and  the  frequency 
resolution  to  about  1 Hz.  The  tine  bandwidth  product  of  a CCD 
transversal  filter  is  siaply  equal  to  the  nuaber  of  stages  H,  i.e. 

B x Td  * fc  x 

c 

where  fc  is  the  clock  frequency.  To  iaplenent  the  conventional  CZT, 
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the  filters  are  weighted,  using  the  split  electrode  technique 
(discussed  in  Section  4.1),  to  fors  a chirp  froe  -f  to  ♦fc  and  the 
preaultiplying  wavefon  has  a duration  I/fc  which  chirps  fros  dc  to 
-fc.  For  a single  frequency  input  at  frequency  fj  the  output  of  the 
first  sixer  sweep  fcoa  -f c to  vfj.  This  wavefora  is  shifted  through 
the  filter  and  forss  a correlation  peat  at  tise 


t 


1 


(4.2-7) 


Thus  the  frequencies  coaponents  appear  tise  ordered  at  the  processor 

output,  the  proportionality  constant  betweeu  frequencies  and  tise 

2 

being  H/fc  , the  filter  chirp  rate.  The  final  post  aultiplication 
siaply  corrects  the  phase  of  the  spectral  coaponents  and  is  often 
oaitted  when  the  phase  is  not  required. 


The  accuracy  of  the  transforaation  is  liaited  primarily  by  the 
accuracy  of  the  CCD  filter  and  the  aultipliars.  Factors  affecting  the 
filter  response  incluia  linearity,  accuracy  of  weighting  technique  and 
theraal  noise  generation. 


CCD  chirp  transfora  processors  have  been  iapleaanted  at  100  kHz 
with  frequency  resolution  of  200  Hz.  The  technique  has  an  advantage 
over  surface  wave  acoustic  iapleaentations  in  that  the  tiaing  can  be 
established  precisely  by  stable  clocking  circuits.  Further,  the  very 
low  insertion  loss  associated  with  the  CCD  process  allows  a dynaaic 
range  of  approxiaataiy  80  dB  to  be  obtained.  The  principle  source  of 
noise  is  the  differential  current  integrator  at  the  output.  Present 
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state  of  the  act  indicates  that  the  chirp  transfora  process  aight  be 

3 

operated  at  20  MHz  with  10  resolution  elements.  The  inaccuracies 
included  in  the  aask  pattern  used  to  generate  the  weighting  liaits  the 
aaplitude  resolution  to  about  0. It.  The  nonunifora  etching  rate  and 
oxide  thickness  togather  with  aask  aisalignaents  contribute  to  an 
overall  error  of  0.5*.^  The  largest  CCD  chirp  transfora  processor 
reported  to  date  a as  500  stages  <14>  and  a transfer  efficiency  of 
0.9999.  At  present  the  chirp  signals  are  generated  off  chip  in  a 8 
bit  BOM  and  the  aultiplier  is  a discrete  aultiplying  D/A.  The 

accuracy  is  liaited  by  the  dyuanic  range  in  the  input  and  output 
circuitry  and  nonlinearities  in  the  aultiplying  circuits.  Further, 

the  dispersion  caused  by  the  finite  transfer  inefficiency  of  1 part  in 

-4  2 

10  causa  a distortion  tera  of  aaplitude  (MPe/2)  tines  the  desired 

response. 

In  concept  all  the  functions  involved  in  the  process  can  be 
accoaplished  by  CCDs.  For  exaaple  the  aultiplication  can  be 
accoaplishad  by  the  CCD  aultiplier  D/A  concept  discussed  in  Section 
4.2  and  the  aultiplying  chirps  aight  bo  stored  in  CCD  aeaory.  By 

using  electron  bean  exposure  the  accuracy  of  tha  aultiplying  D/A  night 
be  aade  equivalent  to  10  bits  plus  sign  and  integrated  on  the  chirp. 
However,  a capable  NOS  BOH  would  be  a wore  effective  technique  since 
the  CCD  aeaory  is  basically  a dynaaic  concept  and  subject  to 

degradation.  The  speai  of  the  process  is  basically  liaited  by  the 
capacitive  area  and  hance  an  increase  in  resolution  to  say  1 Ua  fron 
the  present  5ua  night  lead  to  a systea  operating  at  about  TV  rates. 

i 
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4.3  Beal  Tiae  Nonlinaar  Optical  Data  Processing 

During  January  1975,  we  began  a study  of  real  tiae  nonlinear 
optical  data  processing.  He  are  using  a hybrid  field  effect  liguid 
crystal  light  salsa  <15>  as  an  incoherent  to  coherent  iaaging 
intarface.  This  device  is  used  as  a real  tiae  substitute  for 
hard-clipping  fila  to  accoaplish  pulse  width  encoding  of  continuous 
tone  picture  inforaation  <16>. 

Tha  light  salse  is  used  in  the  coharent  optical  data  processing 
3ystea  shown  in  figura  1.  The  incoherent  input  light  derived  froa  a 
aercury  arc  laap  filtarad  at  a center  waselength  of  525  na.  This 
light  is  aodulated  by  a continuous  tone  transparency  in  contact  with  a 
grating  designed  for  aach  nonlinear  operation.  Tha  experiments  to 
date  hase  been  concerned  with  the  nonlinear  operation  of  iaage  level 
slicing.  The  appropriate  grating  for  this  operation  is  a Bonchi 
ruling  <16>.  The  Bonchi  grating  used  has  a fuadaaental  spatial 
freguency  of  12  cy/ss  and  optical  density  levals  of  0.21  and  0.68. 
Tha  grating  nodulated  scene  is  inaged  onto  the  light-sensitive  CdS 
Layer  of  the  light  vaLve.  The  resultant  nodulation  of  the  CdS 
conductivity  produces  a modulation  of  the  coherent  light  amplitude 
reflected  from  the  ligat  valve.  (For ‘a  detailed  description  of  the 
light  valve  structure  and  function,  the  reader  should  see  reference 
<15>).  The  coherent  light  source  is  a 50  nil  He-Ne  laser. 

The  set-up  of  figure  1 approzinated  the  level-slicing  experiaent 
of  Dashiall  and  Sawchuh  (see  referanca  <1 6>) . The  desired 


input/output  intensity  relationship  for  the  light  valve  has  a high 
gaaaa.  This  high  gaaaa  hard-clips  the  grating  nodulated  iaage.  The 
resultant  coherent  light  output  is  then  pulse  width  nodulated  into 
three  distinct  states:  no  aodulation  (dark) , square  wave  aodulation  at 
the  Honchi  ruling  spatial  frequency  and  no  aodulation  (bright) . 
Band-pass  spatial  filtering  of  the  spatial  frequencies  around  the  t 
first  diffraction  order  of  the  grating  produces  an  output  iaage  of  the 
input  transparency.  This  output  iaage  shows  only  a narrow  slice  of 
the  total  density  range  of  the  input.  The  density  value  for  the 
selected  level  is  detarained  by  the  intensity  of  the  noncoherent  input 
illuaination  to  the  liyht  valve. 

Figure  2 shows  tae  actual  input/output  intensity  relationship 
achieved  with  the  lijat  valve  used  in  the  experiments  to  date.  These 
data  were  recorded  with  a (typical)  light  valve  drive  of  k.O  volts  ras 
at  2.0  kHz.  Figure  2 shows  that  this  valve  has  a aaxiaua  gaaaa  of 
2.1. 


Figure  3 shows  the  transfer  function  achieved  in  the  level 
slicing  experisent.  The  gaaaa  of  the  light  valve  is  too  low  to 
achieve  an  ideal  leval  slice.  However#  a broad  level  slice  is 
observed.  The  filtered  first  order  intensity  increases  to  a peak 
valne  corresponding  to  aaxisus  aodulation  of  the  raflected  coherent 
light.  Bayond  that  path#  the  first  order  output  intensity  decreases. 
This  decrease  is  cnusad  by  ssturation  of  the  light  valve  at  high  input 
intensities. 
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The  seusitoaetry  of  the  light  valve  depends  on  the  drive  voltage 
and  frequency.  This  dcive  can  be  varied  to  aaxiaize  the  gaaaa  of  the 

e 

light  valve.  Other  possible  techniques  for  increasing  the  gaaaa 
include  aodificatioa  of  the  liquid  crystal  type,  the  alignment  of  the 
liquid  crystal  aoleculss  relative  to  the  substrate  surfaces,  the 
preparation  of  the  photoconductor  and  the  operating  teaperature. 

Even  the  liaited  gaaaa  observed  in  these  prelininary  experiments 
can  provide  very  useful  nonlinear  processing.  The  particular  response 
of  the  light  valve  can  be  included  in  the  design  of  the  grating 
aodulator,  for  aonotonic  nonlinearities.  The  aonotonic  nonlinearities 
are  iapleaented  by  spatial  bandpass  filtering  around  the  zero  order  of 
the  grating.  This  approach  allows  the  eztreaely  useful  technique  of 
the  logarithaic  filtering,  as  daaonstrated  by  Kato  and  Goodaan  <17>. 

In  continuing  work  on  this  prograa,  we  plan  to  pursue  two  types 

of  ezperiaents.  The  first  is  to  study  transfer  functions  of  the  light 

« 

valve  under  various  conditions.  This  will  provide  inforaation  on  what 
gaaaas  can  be  produce!  with  the  standard  light  valve.  In  the  second 
3et  of  ezperiaents,  we  will  study  the  quantitative  perforaance  of  the 
light  valve  in  producing  aonotonic  nonlinear  response  functions. 
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